ArXiv: 2310.17022

🎯 Pitch

Need to align a frozen language model with a reward function without touching its weights—and do it for multiple objectives? Controlled Decoding (CD) solves this by training a lightweight prefix scorer that can be combined and transferred across models: it provably matches the optimal RL policy while achieving ~8× sample efficiency gains over best-of-K.

1. Executive Summary

This paper introduces controlled decoding (CD), a modular inference-time alignment framework that solves a tokenwise KL-regularized reinforcement learning objective by training a separate prefix scorer—a value function that predicts the expected future reward from any partially decoded sequence—and then using it to steer generation from a frozen base model, either tokenwise (adjusting logits at each decoding step) or blockwise (selecting the best among K candidate continuations of M tokens). CD provably samples from the optimal RL policy and achieves up to ~8× reduction in sample complexity over standard best-of-K (matching best-of-50 performance with only K=6 on length control), while establishing that the prefix scorer transfers to unseen base models without retraining and that multiple reward scorers can be linearly combined at inference time with no additional training—a capability unavailable to training-time methods like PPO or DPO.

2. Context and Motivation

The Core Problem: How Do We Steer a Frozen Language Model Without Touching Its Weights?

The fundamental question this paper tackles is deceptively simple: if you have a pre-trained language model whose weights you cannot or do not want to modify, how do you make it produce outputs that score highly on some reward function—like being helpful, harmless, concise, or factual—while keeping the model's general capabilities intact?

This matters for at least three practical reasons the paper identifies throughout its introduction and framing. First, a single base model often serves many use cases: a company might deploy one large language model across products with different alignment requirements (chat safety, summarization quality, response length), and retraining the model for each combination of objectives is computationally prohibitive. Second, base models evolve frequently: when a new checkpoint becomes available, training-time alignment methods must be rerun from scratch, whereas modular approaches that treat the base model as frozen could transfer cleanly. Third, alignment objectives are not static: what counts as "helpful" or "safe" changes with context, user preferences, and evolving societal norms, creating a need for inference-time configurability that generator-improvement methods simply cannot provide.

This gap—between the desire for fine-grained control and the reality of frozen, expensive-to-retrain models—is the paper's primary motivation. The authors do not frame this as merely a computational convenience; they argue it is a fundamental requirement for deploying language models responsibly and cost-effectively at scale.

The Two Camps of Alignment Research and the Tradeoff They Present

The paper organizes existing alignment methods into a binary taxonomy that exposes a stark tradeoff (Section 1):

Generator improvement solutions (also called training-time interventions) update the model weights to align the generator with a reward model. The canonical example is KL-regularized PPO (Christiano et al., 2017; Ouyang et al., 2022), where a language model is fine-tuned via reinforcement learning with a KL penalty that prevents it from drifting too far from its initial behavior. Other methods in this category include Direct Preference Optimization (DPO; Rafailov et al., 2023), Sequence Likelihood Calibration (SLiC; Zhao et al., 2022), and Identity Preference Optimization (IPO; Azar et al., 2023). These methods share a common advantage: at inference time, they are efficient—the aligned model simply generates from its updated weights with no additional computation or complexity. But they share a corresponding disadvantage: they offer little to no configurability after training. If you want to change the reward function (e.g., shift from prioritizing helpfulness to prioritizing conciseness, or combine both), you must retrain. If the base model is updated, you must retrain. If a user wants personalized alignment, you must maintain separate models or retrain.

Inference-time add-on solutions (also called controlled generation/decoding) keep the base model frozen and apply an external mechanism at generation time to steer outputs toward high-reward outcomes. The simplest and most widely used is best-of-K (Nakano et al., 2021; Stiennon et al., 2020; Touvron et al., 2023): generate K independent complete responses from the base model, score each with a reward model, and return the highest-scoring one. This is modular (the reward model can be swapped), configurable (K can be adjusted per query), and requires no training of the generator. But it has two critical weaknesses the paper explicitly identifies (Section 3.2):

  1. Latency: all K sequences must be fully decoded before any can be served, which is unacceptable for long-form generation (e.g., writing an essay) or streaming applications.
  2. Sampling inefficiency: achieving high rewards often requires unreasonably large K, because the method evaluates only complete trajectories—it cannot intervene early when a generation is clearly going off-track.

Other inference-time methods exist, most notably FUDGE (Yang & Klein, 2021), which trains a prefix scorer to predict whether a partially decoded sequence will eventually satisfy a desired attribute, then uses that scorer to adjust token-level probabilities during generation. FUDGE is modular and operates tokenwise, giving finer-grained control than best-of-K. However, the paper identifies a critical gap in FUDGE's theoretical foundation: it was not formulated as a solution to a well-defined optimization problem. It lacked a connection to reinforcement learning, making it unclear what objective—if any—FUDGE was optimizing, whether it could be improved, or how it related to training-time methods like PPO.

Other controlled generation methods exist in this space—GeDi (Krause et al., 2021), DIRECTOR (Arora et al., 2022), NADO (Meng et al., 2022), COLD (Qin et al., 2022)—each with their own control mechanisms and divergence constraints. But the paper argues these methods share a common limitation: the absence of a rigorous connection between the inference-time control mechanism and the KL-regularized RL objective that underlies modern alignment. This makes it difficult to understand whether a given inference-time method is doing something principled or merely heuristically reweighting logits, and whether it can provably sample from the optimal aligned distribution.

Where Existing Approaches Fall Short: Specific Gaps

The paper identifies four concrete limitations in the landscape that it aims to address:

1. No clear optimization story for prefix-scorer methods. FUDGE works empirically, but why? The prefix scorer is trained to predict whether a partial sequence will be judged positively by some attribute classifier, and then the base model's logits are adjusted by this prediction. But what is being optimized? The paper notes that prior work left unanswered: is FUDGE solving a KL-regularized RL problem? If so, under what conditions? Without this formalization, it is impossible to understand when FUDGE will succeed or fail, how to improve its training, or how it relates to methods like PPO and DPO that explicitly optimize such an objective.

2. Best-of-K is simple and effective but impractical at scale. The paper cites evidence from Gao et al. (2023) and Rafailov et al. (2023) showing that best-of-K consistently achieves better reward-vs-KL tradeoffs than KL-regularized PPO—a finding that might surprise practitioners who assume training-time optimization is inherently superior. In fact, Yang et al. (2024) provided theoretical reasoning showing that best-of-K is almost optimal for the KL-regularized RL objective. The problem is not the quality of best-of-K's output, but its operational constraints: generating and fully decoding K sequences before serving any response creates untenable latency for long generations, and the K required for high rewards can be impractically large. The paper asks: can we design a method that retains best-of-K's excellent reward-KL tradeoffs while overcoming its latency and sample-efficiency limitations?

3. Training-time methods cannot adapt to new objectives or new base models without retraining. This is the obvious tradeoff the paper exploits, but it is worth being precise about the practical implications. In a production setting where a) the base model is updated weekly, b) different applications need different tradeoffs between, say, helpfulness and conciseness, and c) the alignment targets themselves evolve as safety requirements change, training-time methods require retraining for every combination. This is not merely expensive—it creates a deployment bottleneck where alignment lags behind model improvements. The paper explicitly frames CD as addressing this bottleneck: "We show that the benefits of applying CD transfer to an unseen base model with no further tuning" (Abstract).

4. Multi-objective alignment currently requires retraining. If a deployed system needs to balance helpfulness, harmlessness, and response length, training-time methods must bake these tradeoffs into a single reward function before training. Changing the tradeoff (e.g., making responses more concise while maintaining helpfulness) requires retraining from scratch. The paper argues this is needlessly inflexible and demonstrates that CD's modular design—where prefix scorers for different rewards can be trained independently and combined at inference time via linear combination—solves the multi-objective RL problem with zero additional training (Experiment 4).

How This Paper Positions Itself

The paper's positioning is ambitious but carefully scoped: it does not propose CD as yet another inference-time control method in a crowded field. Instead, it argues that CD provides a unifying framework that connects inference-time controlled generation to the same KL-regularized RL objective that underlies training-time methods like PPO and DPO. The key intellectual move is the following:

The paper starts from the observation that KL-regularized RL has a closed-form optimal policy (Theorem 2.1): the optimal tokenwise decoding distribution is proportional to the base model's distribution multiplied by an exponential term involving the value function—the expected future reward from each possible next token. This means that if you can learn the value function V⋆([x,yt,z])V^\star([x, y_t, z]) for every partial sequence yty_t and candidate next token zz, you can sample from the exact optimal RL policy without ever updating the base model. The problem reduces to learning this value function.

This reframes controlled decoding from "adjusting logits based on some heuristic attribute score" to "learning a value function that, when combined with the base model via Equation 6, provably samples from the KL-regularized RL optimum." The paper then shows that two different training procedures—CD-FUDGE (on-policy, rolling out the base model) and CD-Q (off-policy, using Bellman backups)—both converge to the true value function (or a stationary point thereof) under appropriate assumptions (Theorem 3.1 and the subsequent discussion).

This positioning is significant for two reasons. First, it elevates FUDGE from a heuristic to a principled method by proving that, when trained on base-model rollouts, FUDGE solves the KL-regularized RL problem—a connection that Yang & Klein (2021) did not make. Second, it provides a theoretical foundation for inference-time alignment that parallels the foundation training-time methods have in RL: both are solving the same objective, but CD does so by learning a value function and using it during decoding rather than by updating the generator. This creates a clean conceptual separation: training-time methods improve the proposal distribution (the generator), while CD improves the targeting (the value-function-guided selection), and both are valid approaches to the same RL problem.

The paper is careful not to claim that CD is universally superior to training-time methods. Rather, it argues that CD offers a different set of tradeoffs—flexibility, transferability, and modularity at the cost of additional inference computation—that make it the right choice for many deployment scenarios, especially those involving multiple objectives, frequently updated base models, or the need for per-request configurability. The blockwise CD variant is explicitly positioned as bridging between best-of-K and tokenwise RL, combining best-of-K's excellent reward-KL tradeoffs with lower latency (only M tokens of lookahead rather than full sequence decoding) and better sample efficiency.

Finally, the paper positions CD within the broader landscape of value-based RL for language, drawing explicit connections to DQN (Mnih et al., 2013) for the training procedure and noting that improvements from the deep RL literature (e.g., Rainbow; Hessel et al., 2018) could be applied to further improve CD-Q—though the paper itself uses the simplest form and leaves such extensions to future work. This grounding in the RL literature distinguishes CD from prior controlled generation methods that were developed within the NLP community without explicit connections to value function learning.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

Controlled decoding is a system that steers a frozen language model's output toward desired attributes—like being more helpful, less harmful, or producing shorter responses—by attaching a separate neural network (the prefix scorer) that judges the quality of partial generations and reweights the base model's token probabilities during decoding, without ever modifying the base model's weights. The system solves the problem of aligning language model outputs to reward functions while preserving the base model's general capabilities, using a modular design where the prefix scorer can be trained once and then combined with different base models, different reward functions, or even linear combinations of multiple rewards at inference time with zero additional training.

3.2 Big-picture architecture (diagram in words)

The system has five major components that interact at both training time and inference time:

  1. Base language model (πref\pi_{\text{ref}}): a frozen pre-trained autoregressive LM (PaLM 2-XXS in experiments) that provides the proposal distribution over tokens—the raw probabilities from which all generations originate. This model is never updated by CD.

  2. Prefix scorer (VθV_\theta): a neural network (fine-tuned from the same PaLM 2-XXS architecture) that takes a prompt xx concatenated with a partially decoded response yty_t of tt tokens and outputs a scalar value estimating the expected future reward from continuing generation from that point using πref\pi_{\text{ref}}. There are two training variants: CD-FUDGE trains on-policy using completed rollouts from the base model; CD-Q trains off-policy using Bellman backups (temporal difference learning).

  3. Reward function (r([x,y])r([x, y])): a scalar function that scores fully completed sequences. The paper experiments with three: response length (log of token count), helpfulness/harmlessness (a Bradley-Terry reward model trained on Anthropic HH pairwise preferences), and summarization quality (a similar reward model trained on TL;DR preferences).

  4. Inference-time sampling strategy: two mechanisms for using the prefix scorer during generation:

    • Tokenwise sampling: at each decoding step, the base model's logits are combined with the prefix scorer's value estimates for each candidate next token (via an exponential scaling controlled by λ\lambda), and the next token is sampled from this reweighted distribution. This requires one prefix scorer call per generated token.
    • Blockwise best-of-K: at each block boundary, KK candidate continuations of MM tokens are sampled independently from the base model, the prefix scorer evaluates each complete block, and the highest-scoring block is selected as the continuation. This requires KK parallel decodes per block but only MM tokens of lookahead latency.

  5. Multi-objective combiner (inference-time only): when multiple rewards are desired, the prefix scorers for each reward are evaluated independently and their outputs are linearly combined (with configurable weights) to produce a single score used in either tokenwise or blockwise sampling. No training is required for new combinations.

Information flow at training time: For CD-FUDGE, the base model generates complete responses to prompts → the reward function scores each complete response → the prefix scorer is trained to predict this final reward from each partial prefix of the response, using MSE loss. For CD-Q, an off-policy dataset of prompt-response pairs is scored by the reward function → the prefix scorer is trained to satisfy the Bellman equation: at non-terminal states, the predicted value should equal the expected value of the next state under πref\pi_{\text{ref}}; at terminal states, it should equal the reward.

Information flow at inference time: A prompt xx enters → the base model and prefix scorer are both loaded → the sampling strategy (tokenwise or blockwise) iteratively generates tokens → for tokenwise, at each step the prefix scorer evaluates all candidate next tokens and reweights the base logits → for blockwise, periodically KK candidate blocks are sampled and scored, with the best one accepted → the process continues until an EOS token is generated → the complete response is returned.

3.3 Roadmap for the deep dive

  • First, the formal KL-regularized RL objective (Equation 2) and its closed-form optimal policy (Theorem 2.1), because the entire approach follows directly from this mathematical result—understanding why the optimal policy has the form p(z)eλV⋆p(z) e^{\lambda V^\star} is prerequisite to understanding why CD works at all.
  • Second, the value function V⋆V^\star and the training procedures for estimating it (CD-FUDGE and CD-Q), since the prefix scorer is the learned approximation of this value function and the training methodology determines when and whether CD provably samples from the optimal RL policy.
  • Third, the two inference-time sampling strategies (tokenwise and blockwise) and their operational tradeoffs, because CD is not a single algorithm but a family of methods parameterized by how the prefix scorer is used during generation.
  • Fourth, the connection between blockwise CD and best-of-K, including the KL divergence bounds for blockwise sampling and the argument for why blockwise CD achieves best-of-K's excellent reward-KL tradeoffs with lower latency and better sample efficiency.
  • Fifth, the multi-objective extension and base-model transfer capabilities, which are the features that distinguish CD most sharply from training-time methods.
  • Sixth, the concrete training configurations, hyperparameters, and computational budget, to make the approach reproducible.

3.4 Detailed, sentence-based technical breakdown

This is primarily a theoretical framework paper with strong empirical validation whose core idea is that the KL-regularized RL alignment objective has a closed-form optimal decoding policy expressible entirely in terms of the base model and the value function, and that learning this value function via prefix-scorer training enables modular, inference-time control that is provably equivalent to training-time RL while offering superior flexibility.

The KL-Regularized RL Objective and Its Closed-Form Solution

The paper begins by formalizing the alignment problem as a tokenwise KL-regularized reinforcement learning objective. This is not the standard sequence-level objective used in RLHF and DPO, but a per-token version that enables fine-grained, step-by-step control. Understanding why this tokenwise formulation produces the closed-form solution in Theorem 2.1 is the conceptual foundation for everything that follows.

The reward model. Let xx be a prompt and y=yT:=[y1,…,yT]y = y_T := [y_1, \ldots, y_T] be a response of TT tokens, where each yt∈Yy_t \in \mathcal{Y} (the vocabulary). The paper defines a scalar reward function r([x,y])r([x, y]) that scores complete prompt-response pairs. This reward is bounded from above—a technical condition that ensures the value function is well-behaved. The paper uses a tokenwise reward R([x,yt])R([x, y_t]) that is zero for all intermediate tokens and equals r([x,yt])r([x, y_t]) only when yty_t is the end-of-sequence token:

\begin{cases} 0 & y_t \neq \text{EOS} \\ r([x, y_t]) & y_t = \text{EOS} \end{cases}$$ where EOS represents the end of sequence. **What this means:** the system only receives reward after generating a complete response. During intermediate decoding steps, no reward signal is available. This mirrors the standard RLHF setup where the reward model evaluates full completions. The purpose of defining this tokenwise reward—even though it's zero everywhere except at the terminal step—is to set up a per-step value function that can guide generation before the final outcome is known. **The value function.** The value function $V^\star([x, y_t])$ is defined as the expected cumulative future reward when continuing generation from a partially decoded sequence $y_t$ using the base language model $\pi_{\text{ref}}$: $$V^\star([x, y_t]) := \mathbb{E}_{z_1, z_2, \ldots \sim \pi_{\text{ref}}} \left\{ \sum_{\tau \geq 0} R([x, y_t, z_\tau]) \right\}$$ where $z_1, z_2, \ldots$ are the future tokens sampled autoregressively from $\pi_{\text{ref}}$ to complete the response. **What it computes:** for any partial sequence, this is the expected reward that the base model would eventually achieve if allowed to continue generating from that point. It is a forward-looking estimate: "if I'm at this partial response and the base model takes over from here, what reward do I expect to get on average?" Since the reward is only non-zero at EOS, this is essentially the expected value of $r([x, y])$ for completions from $\pi_{\text{ref}}$ starting from $[x, y_t]$. **Why this form:** the value function captures all the information needed to decide whether a partial sequence is on a promising trajectory. If $V^\star$ is high, continuing from this point tends to lead to good outcomes; if low, it tends to lead to poor outcomes. This is the standard definition from reinforcement learning (Sutton & Barto, 2018), but applied to the specific case of autoregressive text generation where the state is the partial sequence and actions are token choices. **The advantage function.** For any decoding policy $\pi$ (which could be different from $\pi_{\text{ref}}$), the tokenwise advantage is: $$A([x, y_t]; \pi) := \mathbb{E}_{z \sim \pi} \left[ V^\star([x, y_t, z]) - V^\star([x, y_t]) \right]$$ where the expectation is over the next token $z$ drawn from policy $\pi$. This expands to: $$A([x, y_t]; \pi) = \sum_{z \in \mathcal{Y}} \pi(z|[x, y_t]) V^\star([x, y_t, z]) - V^\star([x, y_t])$$ **What it computes:** how much better (or worse) the expected future reward is when choosing the next token according to $\pi$ versus the current value. It answers the question: "does deviating from $\pi_{\text{ref}}$ at this step increase or decrease my expected final reward?" **Why this form:** when $\pi = \pi_{\text{ref}}$, the advantage is exactly zero by the law of total probability—the expected next-state value under the base policy equals the current value. This provides a clean baseline: a positive advantage means the alternative policy $\pi$ improves over the base model; a negative advantage means it worsens outcomes. The goal of alignment is to find a policy that achieves positive advantage without deviating too far from $\pi_{\text{ref}}$. **The KL-regularized RL objective.** The paper defines the per-token objective that trades off advantage against deviation: $$J_\lambda([x, y_t]; \pi) := \lambda A([x, y_t]; \pi) - D([x, y_t]; \pi)$$ where $\lambda \in \mathbb{R}_{\geq 0}$ is the tradeoff parameter and $D([x, y_t]; \pi)$ is the tokenwise KL divergence between $\pi$ and $\pi_{\text{ref}}$: $$D([x, y_t]; \pi) := \text{KL}(\pi(\cdot|[x, y_t]) \| \pi_{\text{ref}}(\cdot|[x, y_t])) = \sum_{z \in \mathcal{Y}} \pi(z|[x, y_t]) \log \left( \frac{\pi(z|[x, y_t])}{\pi_{\text{ref}}(z|[x, y_t])} \right)$$ **What it computes:** a scalar objective that the decoding policy should maximize at each step. The first term ($\lambda A$) pushes the policy toward tokens that lead to higher expected reward. The second term ($-D$) penalizes deviation from the base model, acting as a regularizer that prevents the policy from degenerating into a reward-maximizing but incoherent distribution. When $\lambda = 0$, the optimal policy is $\pi_{\text{ref}}$ (no deviation). As $\lambda$ increases, the policy is allowed to deviate more to chase reward. **Why this form:** this is the tokenwise analog of the sequence-level KL-regularized objective used in RLHF. The paper notes that $J_\lambda$ is concave in $\pi$ because $A$ is linear in $\pi$ and $-D$ is concave (KL divergence is convex, so its negative is concave). Concavity guarantees a unique global maximum, which is essential for the closed-form solution. **The main theoretical result.** Theorem 2.1 states that the unique optimal policy that maximizes $J_\lambda$ at every step is: $$\pi^\star_\lambda(z|[x, y_t]) \propto \pi_{\text{ref}}(z|[x, y_t]) e^{\lambda V^\star([x, y_t, z])}$$ The proportionality constant is the normalization factor $Z_\lambda([x, y_t]) = \sum_{z \in \mathcal{Y}} \pi_{\text{ref}}(z|[x, y_t]) e^{\lambda V^\star([x, y_t, z])}$. **What this means in plain language:** to sample from the optimal aligned policy, you don't need to train a new generator. You take the base model's probability for each candidate next token, multiply it by $e^{\lambda \times \text{(value of the resulting partial sequence)}}$, renormalize, and sample. Tokens that lead to higher-value future states get upweighted; tokens that lead to lower-value future states get downweighted. The strength of this reweighting is controlled by $\lambda$. **Why this form matters:** this result is the entire intellectual foundation for controlled decoding. It says that the alignment problem reduces to learning the value function $V^\star$. If you can learn a good approximation of $V^\star$, you can implement the optimal RL policy at inference time by simply reweighting the base model's logits with the exponentiated value function—exactly what CD does. The paper draws an explicit connection to Korbak et al. (2022), who observed a similar relationship between KL-regularized RL and Bayesian inference, but notes that the key difference here is that the controller operates tokenwise rather than sequence-level, enabling step-by-step intervention. **Proof sketch.** The paper provides the proof in Appendix C and it proceeds as follows (the explanation here is paraphrased from the proof): The objective $J_\lambda$ can be rewritten as the negative KL divergence between $\pi$ and a target distribution $q_\lambda$ plus a constant $\log Z_\lambda$: $$J_\lambda([x, y_t]; \pi) = -\text{KL}(\pi(\cdot|[x, y_t]) \| q_\lambda(\cdot|[x, y_t])) + \log Z_\lambda([x, y_t])$$ where $q_\lambda(z|[x, y_t]) = \frac{\pi_{\text{ref}}(z|[x, y_t]) e^{\lambda V^\star([x, y_t, z])}}{Z_\lambda([x, y_t])}$. Since KL divergence is non-negative and equals zero only when the two distributions are identical, the unique maximizer of $J_\lambda$ is $\pi^\star_\lambda = q_\lambda$. The proof relies on rearranging the objective to isolate the KL term, which is possible because the advantage is linear in $\pi$ and the KL penalty provides the convex regularization that makes the objective concave. --- #### Training the Prefix Scorer: CD-FUDGE The core practical challenge is learning a function $V_\theta([x, y_t])$ parameterized by $\theta$ that approximates the true value function $V^\star([x, y_t])$. The paper presents two training methods. CD-FUDGE is the simpler, on-policy approach. **Training data generation.** For CD-FUDGE, training data is generated by rolling out the base model. Specifically, given a prompt $x \sim \mu$ (where $\mu$ is a distribution over training prompts), a complete response $y = [y_1, \ldots, y_T]$ is sampled stochastically from $\pi_{\text{ref}}(\cdot|x)$. The reward $r([x, y])$ is computed for the complete response—this could be the response length, a classifier score, or the output of a learned reward model. **The CD-FUDGE loss function.** The prefix scorer is trained to predict, from each prefix $y_t$ (for $t \in [|y|]$), the final reward $r([x, y])$ that the complete response ultimately received. The loss is mean squared error averaged over all prefixes and all training examples: $$\mathcal{L}_F(\theta) = \mathbb{E}_{x \sim \mu} \ell_F(x, y; \theta), \quad \text{s.t. } y \sim \pi_{\text{ref}}$$ where $$\ell_F(x, y; \theta) = \frac{1}{2} \sum_{t \in [|y|]} \left( V_\theta([x, y_t]) - r([x, y]) \right)^2$$ **What it computes:** for each partial prefix in a generated response, the prefix scorer is trained to output the scalar value that equals the reward the complete response eventually achieved. The sum is over all prefix positions $t$ in the response. The factor $\frac{1}{2}$ is conventional for MSE. **Why this works.** The key theoretical result (Theorem 3.1 and Lemma C.1 in Appendix C) is that stochastic gradient descent on $\mathcal{L}_F$ converges to a stationary point of the true value function objective $\mathcal{L}^\star(\theta)$: $$\mathcal{L}^\star(\theta) = \mathbb{E}_{x \sim \mu} \mathbb{E}_{y \sim \pi_{\text{ref}}(\cdot|x)} \frac{1}{2} \sum_{t \in [|y|]} \left( V_\theta([x, y_t]) - V^\star([x, y_t]) \right)^2$$ The proof (Lemma C.1) shows that the gradient of $\mathcal{L}_F$ is an unbiased estimator of the gradient of $\mathcal{L}^\star$: $$\mathbb{E}_{y \sim \pi_{\text{ref}}}[\nabla_\theta \mathcal{L}_F(\theta)] = \nabla_\theta \mathcal{L}^\star(\theta)$$ The intuition is as follows. The CD-FUDGE loss trains on pairs (prefix, final reward) where the prefix is from the base model's rollout and the target is the actual reward of that rollout. The expected value of the final reward given the prefix is exactly $V^\star([x, y_t])$—this follows from the definition of $V^\star$ as the expected future reward under $\pi_{\text{ref}}$. Therefore, although each training example uses a noisy single-sample estimate of the reward, the expectation over rollouts from $\pi_{\text{ref}}$ yields the true value function. The proof works through the algebra of expanding the squared error, applying the law of total expectation to replace $r([x, y])$ with $V^\star([x, y_t])$ in cross-terms, and showing that the constant terms (involving $r^2$) drop out of the gradient. **Why this form matters.** This is a remarkable simplification. It means that CD-FUDGE does not need to know the true value function during training—it only needs rollouts from the base model and their associated rewards. The prefix scorer learns to implicitly average over all possible completions from each prefix simply by seeing many rollouts and regressing on their final rewards. This makes CD-FUDGE simple to implement: generate data with the base model, score it with the reward function, and train a standard regression model. **Practical considerations.** The paper notes that CD-FUDGE requires on-policy data (generated by the base model $\pi_{\text{ref}}$) for the theoretical guarantee to hold. If the data comes from a different distribution, the expectation in Lemma C.1 would not equal the gradient of $\mathcal{L}^\star$. This is why CD-FUDGE is trained on base-model rollouts specifically. The convergence rate depends on standard assumptions (Lipschitz continuity, Polyak-Łojasiewicz inequality) as stated in Theorem C.2. --- #### Training the Prefix Scorer: CD-Q CD-Q is an off-policy alternative that does not require rolling out the base model to generate training data. Instead, it uses temporal difference learning—the same principle behind DQN (Mnih et al., 2013)—to learn the value function from an offline dataset. **The Bellman equation for this setting.** The true value function satisfies a recursive relationship (Bellman equation): $$V^\star([x, y_t]) = \begin{cases} \mathbb{E}_{z \sim \pi_{\text{ref}}(\cdot|[x, y_t])} V^\star([x, y_t, z]) & y_t \neq \text{EOS} \\ r([x, y_t]) & y_t = \text{EOS} \end{cases}$$ **What this says:** for non-terminal states (no EOS yet), the value at a partial sequence equals the expected value after taking one more step according to $\pi_{\text{ref}}$. For terminal states (EOS reached), the value simply equals the reward. This is a consistency condition: the value function should be self-consistent under the base policy's dynamics. **The CD-Q loss function.** The paper optimizes a temporal difference loss that encourages the prefix scorer to satisfy this Bellman equation: $$\mathcal{L}_Q(\theta) = \mathbb{E}_{x \sim \mu} \ell_Q(x, y; \theta)$$ where $$\ell_Q(x, y; \theta) = \frac{1}{2} \sum_{t \in [|y|]} \left( V_\theta([x, y_t]) - \dot{v}_t \right)^2$$ and the target $v_t$ is computed as: $$v_t = \begin{cases} \sum_{z \in \mathcal{Y}} \pi_{\text{ref}}(z|[x, y_t]) V_\theta([x, y_t, z]) & y_t \neq \text{EOS} \\ r([x, y_t]) & y_t = \text{EOS} \end{cases}$$ The notation $\dot{v}_t$ indicates a **stop gradient**: $v_t$ is computed using the current parameters $\theta$, but gradients do not flow through this computation—it is treated as a constant target. **What it computes:** for each prefix $y_t$ in the training data, the loss compares the prefix scorer's prediction $V_\theta([x, y_t])$ with a target $v_t$. For non-terminal prefixes, the target is the expected value of the next state under $\pi_{\text{ref}}$, computed by summing over all possible next tokens $z$ weighted by $\pi_{\text{ref}}(z|[x, y_t])$ and their predicted values $V_\theta([x, y_t, z])$. For terminal prefixes (ending in EOS), the target is simply the reward $r([x, y_t])$. **Why this form:** this is policy evaluation for the base policy—learning the value function of a fixed policy ($\pi_{\text{ref}}$) from data. The stop gradient is critical for stability: without it, the optimization would try to simultaneously adjust $V_\theta$ at all states, creating a moving target problem where the objective changes as parameters update. This is the same technique used in DQN to stabilize training. **Off-policy data.** Unlike CD-FUDGE which requires on-policy rollouts from $\pi_{\text{ref}}$, CD-Q can be trained on any dataset of $(x, y)$ pairs, provided that the reward $r([x, y])$ can be computed for each complete response. The base model $\pi_{\text{ref}}$ is still needed during training to compute the expectation in the target $v_t$ for non-terminal states, but the trajectories themselves need not come from $\pi_{\text{ref}}$. This makes CD-Q more flexible: you can use existing datasets, expert demonstrations, or any source of prompt-response pairs, score them with the reward function, and train the prefix scorer via TD learning. **Convergence properties.** The paper notes that a simple modification of the CD-Q procedure can be shown to be provably convergent, citing Wang & Ueda (2022). The authors also remark that many improvements over DQN exist (collected under the Rainbow framework; Hessel et al., 2018), and exploring how to improve CD-Q using these techniques is left to future work. The paper uses the simplest form of TD learning because "it already gives good empirical performance." **Key difference between CD-FUDGE and CD-Q training.** CD-FUDGE trains the prefix scorer to directly predict the final reward from partial prefixes, using rollouts from $\pi_{\text{ref}}$ to provide unbiased (but high-variance) targets. CD-Q trains the prefix scorer to be self-consistent according to the Bellman equation, using one-step lookahead targets that have lower variance but introduce bias through the use of the current (imperfect) value estimates as targets. The paper's experiments suggest that this difference matters for blockwise sampling: CD-FUDGE's predictions are much noisier than CD-Q's (Figure 13, Appendix B), which explains why blockwise CD-FUDGE fails to match blockwise CD-Q's performance—the noisy value estimates make the blockwise ranking unreliable. **The true value function objective for reference.** Both methods are ultimately trying to minimize the same ideal objective: $$\mathcal{L}^\star(\theta) = \mathbb{E}_{x \sim \mu} \mathbb{E}_{y \sim \pi_{\text{ref}}(\cdot|x)} \frac{1}{2} \sum_{t \in [|y|]} \left( V_\theta([x, y_t]) - V^\star([x, y_t]) \right)^2$$ This is the expected squared error between the learned prefix scorer and the true value function, where the expectation is over prompts and base-model rollouts. The paper mentions in a footnote that "it may be possible to devise a more effective distillation objective through Fisher information shaping or other divergences," acknowledging that MSE is a simple but potentially suboptimal choice. --- #### Tokenwise Sampling at Inference Time Once the prefix scorer $V_\theta$ is trained (via either CD-FUDGE or CD-Q), it is used at inference time to implement the optimal policy from Theorem 2.1. **The tokenwise sampling rule.** Given context $x$ and partially decoded sequence $y_t$, the next token is sampled from: $$z \sim \pi_\theta(\cdot|[x, y_t]) \quad \text{where} \quad \pi_\theta(z|[x, y_t]) \propto \pi_{\text{ref}}(z|[x, y_t]) e^{\lambda V_\theta([x, y_t, z])}$$ **What it computes:** for each candidate next token $z$ in the vocabulary, the system computes two values: the base model's log-probability $\log \pi_{\text{ref}}(z|[x, y_t])$ from the frozen LM, and the prefix scorer's value estimate $V_\theta([x, y_t, z])$ for the partial sequence extended by token $z$. These are combined linearly: $\log \pi_{\text{ref}}(z) + \lambda V_\theta(z)$. The result is exponentiated and normalized to form a probability distribution, and the next token is sampled from this distribution. **What this achieves:** tokens that the base model considers likely AND that lead to high-value future states receive increased probability. Tokens that are likely under the base model but lead toward low-reward outcomes are suppressed. The hyperparameter $\lambda$ controls the strength of the value signal relative to the base model's preferences—at $\lambda = 0$, the distribution is exactly $\pi_{\text{ref}}$; as $\lambda$ increases, the distribution becomes increasingly dominated by the value function. **Why this is the optimal policy:** Theorem 2.1 proved that this reweighting, when using the true value function $V^\star$, exactly maximizes the tokenwise KL-regularized RL objective $J_\lambda$. By substituting the learned $V_\theta$ for $V^\star$, the system approximates this optimal policy. The quality of the approximation depends on how well $V_\theta$ estimates $V^\star$. **Operational characteristics.** Tokenwise sampling requires one call to the prefix scorer for every token in every candidate position in the vocabulary at each decoding step. In principle, this means $|\mathcal{Y}|$ prefix scorer evaluations per generated token, which would be computationally prohibitive for large vocabularies (tens of thousands of tokens). In practice, the paper presumably evaluates the prefix scorer only for a subset of candidates (e.g., the top-$k$ tokens from the base model), though this optimization is not explicitly described in the paper. The cost is linear in the number of generated tokens—each decoding step incurs a constant overhead from the prefix scorer. **The conceptual diagram in Figure 1.** The paper illustrates tokenwise sampling with a sentiment control example: the prompt is "Will this paper get accepted?" and candidate continuations like "This paper will be liked" (high sentiment prefix score) versus "disliked" (low prefix score) are evaluated. The aligned score combines the LM likelihood with the sentiment prefix score, downweighting tokens leading to negative sentiment and upweighting tokens leading to positive sentiment. --- #### Blockwise Best-of-K Sampling at Inference Time Blockwise CD is the paper's second inference-time strategy and the one that empirically performs best. It bridges between tokenwise RL control and the sequence-level best-of-K approach. **The blockwise sampling rule.** At a decoding step with partial sequence $y_t$, the system samples $K$ independent candidate continuation blocks of length $M$ tokens from the base policy: $$\{z^M_{(k)}\}_{k \in [K]} \stackrel{\text{i.i.d.}}{\sim} \pi_{\text{ref}}(z^M|[x, y_t])$$ where $z^M$ denotes a sequence of $M$ tokens. Each candidate block is scored by the prefix scorer $V_\theta([x, y_t, z^M_{(k)}])$. The block with the highest score is accepted as the continuation, and the others are discarded: $$z^M := \arg\max_{\{z^M_{(k)}\}_{k \in [K]}} V_\theta([x, y_t, z^M_{(k)}])$$ The process then advances $M$ tokens and repeats—sampling $K$ new candidate blocks from the new partial sequence—until a candidate containing an EOS token is accepted, at which point the response is complete. **What this achieves structurally:** instead of intervening at every single token (tokenwise CD), the system intervenes every $M$ tokens by selecting among $K$ alternative futures. This creates a tree-structured search with branching factor $K$ and depth determined by the response length divided by block size. Only one branch is pursued at each level—the one with the highest prefix scorer value—making this a greedy best-first search rather than an exhaustive beam search. **The relationship to best-of-K.** Standard best-of-K generates $K$ complete sequences independently, scores them all with a reward model, and returns the best one. Blockwise CD is similar in that it compares $K$ alternatives and selects the best, but with crucial differences: 1. **Latency:** blockwise CD only requires decoding $M$ tokens before a decision can be made, whereas best-of-K requires fully decoding all $K$ sequences (potentially hundreds of tokens) before any can be returned. This makes blockwise CD suitable for streaming and long-form generation. 2. **Intervention granularity:** blockwise CD corrects course every $M$ tokens rather than only at the end. If a generation starts going off-track, it can be redirected at the next block boundary rather than being committed to a poor trajectory for its entire length. 3. **Sampling efficiency:** because blockwise CD intervenes multiple times during generation (every $M$ tokens), it can achieve high rewards with smaller $K$ than best-of-K, which only gets one selection decision per complete response. The paper demonstrates this empirically: blockwise CD-Q with $K = 6$ matches best-of-K with $K = 50$ on the length control task (Figure 3). **The conceptual diagram in Figure 2.** The paper illustrates blockwise sampling with the same sentiment example: four candidate continuations of four tokens each are sampled ("This paper will be liked by", "will receive diverging reviews", "may be liked by", "is not getting into"), each is scored by the sentiment prefix scorer, and the highest-scoring block is selected. **KL divergence bound for blockwise CD.** The paper provides an upper bound on the KL divergence between the blockwise CD policy and the base policy, extending the known bound for best-of-K (Beirami et al., 2024, Theorem 1): $$\text{KL}(\pi \| \pi_{\text{ref}}) \leq \mathbb{E}_{x \sim \mu} \left( \log(K) - \frac{K-1}{K} \right) \left\lceil \frac{L_x}{M} \right\rceil$$ where $L_x$ is the total number of decoded tokens in the full response for prompt $x$, and $\lceil L_x / M \rceil$ is the number of blockwise selection steps. This bound is intuitive: at each block boundary, selecting the best of $K$ samples incurs a KL cost of at most $\log(K) - (K-1)/K$, and this cost is incurred once per block, so the total KL is approximately that per-block cost times the number of blocks. --- #### Why Blockwise CD Outperforms Tokenwise CD Empirically This is one of the paper's most important empirical findings and requires understanding the gap between theory and practice. **The theoretical expectation.** Tokenwise CD (with perfect value estimates) provably samples from the optimal policy for the tokenwise KL-regularized RL objective. Therefore, in theory, tokenwise CD should achieve the best possible reward-vs-KL tradeoff. Blockwise CD is a coarser approximation—it only optimizes at block boundaries rather than at every token. **The empirical reality.** Across all experiments (Figures 3, 4, 5), blockwise CD consistently achieves better reward-vs-KL tradeoffs than tokenwise CD, and blockwise CD-Q matches or approaches best-of-K's performance. The paper does not claim to fully explain this phenomenon, but provides context from the literature: - **Gao et al. (2023, Figure 1)** and **Rafailov et al. (2023, Figure 3)** independently observed that best-of-K consistently achieves better reward-KL tradeoffs than KL-regularized PPO, even though PPO directly optimizes the sequence-level RL objective that best-of-K only approximates. - **Yang et al. (2024)** provided theoretical reasoning showing that best-of-K is an almost optimal solution to the KL-regularized RL problem—essentially, the gap between best-of-K's empirical performance and the theoretical optimum is small. **The paper's interpretation.** The authors suggest that the tokenwise RL objective may be too restrictive compared to the sequence-level objective. By optimizing token-by-token, tokenwise CD may get stuck in local optima or fail to make the kind of global tradeoffs that best-of-K and blockwise CD can make by evaluating complete (or block-length) trajectories. Blockwise CD inherits best-of-K's strong empirical properties while adding the operational advantages of lower latency and more frequent intervention points. **The CD-FUDGE vs. CD-Q gap in blockwise performance.** Figure 13 (Appendix B) provides additional insight: when used to predict the final reward of complete responses, CD-Q's predictions are much better aligned with actual rewards than CD-FUDGE's predictions, which are noisy. In blockwise CD, the prefix scorer must accurately rank different candidate blocks—a task that requires good relative value estimates. CD-FUDGE's noisy predictions make this ranking unreliable, explaining why blockwise CD-FUDGE performs poorly while blockwise CD-Q performs well. Tokenwise CD is less sensitive to this noise because it only needs to nudge probabilities, not make hard argmax selections. --- #### Multi-Objective Control Through Linear Combination One of CD's most practically significant capabilities is combining multiple reward objectives at inference time with no additional training. **The mechanism.** If prefix scorers $V^{(1)}_\theta, V^{(2)}_\theta, \ldots, V^{(m)}_\theta$ have been trained independently for $m$ different reward functions $r_1, r_2, \ldots, r_m$, they can be combined at inference time by forming a weighted sum: $$V^{\text{combined}}_\theta([x, y_t]) = \sum_{i=1}^m \alpha_i V^{(i)}_\theta([x, y_t])$$ where $\alpha_i$ are user-specified weights that control the relative importance of each objective. This combined prefix scorer is then used in either tokenwise or blockwise sampling exactly as a single-objective scorer would be. **What this achieves:** the system can, for example, increase helpfulness (positive weight on HH prefix scorer) while simultaneously preventing responses from becoming too verbose (negative weight on length prefix scorer). The user can tune the tradeoff at inference time by adjusting the $\alpha$ weights—no retraining, no new data collection, no model checkpoint management. **Why training-time methods cannot do this.** PPO, DPO, and IPO train the generator to optimize a single, fixed reward function. The reward weights are baked into the training objective and cannot be changed afterward. To achieve a different tradeoff between objectives, these methods must be retrained from scratch with the new reward combination. The paper explicitly notes: "this experiment would be impossible with training-time KL-regularized RL methods (PPO/DPO/IPO) as they need to be retrained from scratch for different linear combinations of rewards" (Section 5, Experiment 4 discussion). **Experimental demonstration (Experiment 4, Figure 6).** The paper demonstrates this on the HH + length control task. Applying only the HH prefix scorer for blockwise decoding improves helpfulness/harmlessness but also increases response length (consistent with prior findings that optimizing for helpfulness often produces verbose outputs). By adding a negative weight on the length prefix scorer, the system can keep response length approximately at baseline levels while still improving HH—at the cost of a slight reduction in HH improvement. The tradeoff curve in Figure 6 shows that different linear combinations trace out a Pareto frontier in the (length, HH win rate) space, all achievable from the same two prefix scorers. --- #### Transfer to an Unseen Base Model Another key property of CD is that the prefix scorer transfers to base models it was not trained with. **Why this works in principle.** The prefix scorer is trained to estimate the value function $V^\star$ for the base policy $\pi_{\text{ref}}$ on which it was trained. If a different base model $\pi'_{\text{ref}}$ has a similar output distribution to $\pi_{\text{ref}}$, then $V^\star$ (the expected reward under $\pi_{\text{ref}}$) may still be a reasonable value estimate for generations from $\pi'_{\text{ref}}$. More precisely, the prefix scorer is learning a function of the partial sequence content, not of which model produced it—if two base models tend to produce similar completions from similar prefixes, the value function transfers. **Why it matters practically.** In production systems, base models are frequently updated (new checkpoints, larger architectures, different training data). If the prefix scorer needed retraining for every new base model, CD would lose much of its modularity advantage. The paper demonstrates transfer from PaLM 2-XXS (the model the prefix scorer was trained with) to PaLM 2-S and PaLM 2-XS, showing that blockwise CD-Q retains its performance on par with best-of-K without any retraining (Figures 7 and 8). **Limitation acknowledged.** The paper does not claim universal transfer. The transfer works between models in the same family (PaLM 2 variants) with presumably similar output distributions. Transfer across dramatically different architectures or capabilities (e.g., from a 100M parameter model to a 100B parameter model) may not work as well, since the value function for one base policy could be a poor estimate for another with very different behavior. The paper leaves systematic study of transfer limits to future work. --- #### KL Divergence Estimation for Evaluation Since the paper evaluates methods by their reward-vs-KL tradeoffs, it needs reliable KL estimates for policies that are defined implicitly through sampling procedures (best-of-K, blockwise CD) rather than through explicit probability distributions. **Best-of-K KL bound.** For standard best-of-K, the paper uses the known upper bound (Stiennon et al., 2020; Beirami et al., 2024): $$\text{KL}(\pi \| \pi_{\text{ref}}) \leq \log(K) - \frac{K-1}{K}$$ This bound is exact in the sense that the inequality is tight under certain conditions, and it is widely used in the literature as a proxy for the true KL divergence when the aligned policy is defined via rejection sampling. **Blockwise CD KL bound.** The paper extends this bound to the blockwise setting. Since blockwise CD makes $\lceil L_x / M \rceil$ independent best-of-K selections (one per block), the total KL is bounded by the per-block bound multiplied by the number of blocks: $$\text{KL}(\pi \| \pi_{\text{ref}}) \leq \mathbb{E}_{x \sim \mu} \left( \log(K) - \frac{K-1}{K} \right) \left\lceil \frac{L_x}{M} \right\rceil$$ where the expectation is over the prompt distribution because different prompts produce responses of different lengths $L_x$, hence different numbers of block selection steps. **Tokenwise CD KL estimation.** For tokenwise CD, the KL can be estimated more directly because the policy $\pi_\theta$ is defined explicitly at each step (it is the reweighted distribution from Equation 6). The sequence-level KL can be computed by summing per-token KL divergences or estimated via sampling, though the paper does not detail the exact procedure. **Practical note on KL values.** The paper focuses on KL values smaller than 10, "beyond which the policy shows significant signs of overfitting" (citing Eisenstein et al., 2023). This is a practical threshold: at very high KL divergences, the aligned policy has deviated so far from the base model that it may produce degenerate or nonsensical outputs even if the reward is high. The goal is to find methods that achieve high reward at low KL cost. --- #### Concrete Training Configurations and Hyperparameters **Base model.** All experiments use PaLM 2-XXS (Anil et al., 2023) as the base generative model. The prefix scorer is also fine-tuned from PaLM 2-XXS, meaning it shares the same architecture and initial weights as the base model. This is a design choice: using the same base architecture ensures the prefix scorer can process partial sequences in the same embedding space as the base model uses for generation. **Reward models for HH and summarization.** The helpfulness/harmlessness reward model is trained by fine-tuning PaLM 2-XXS on the combined Anthropic HH dataset (helpfulness + harmlessness) using the Bradley-Terry pairwise preference model. Training uses a pairwise cross-entropy loss between the model's predicted preferences and human preferences, with learning rate $1 \times 10^{-4}$ for 1 epoch. The checkpoint with the highest evaluation accuracy is selected. The summarization quality reward model follows the same procedure on the TL;DR preference dataset, but with learning rate $1 \times 10^{-5}$ for 1 epoch. **CD-FUDGE training.** Trained via SGD on the objective in Equation 4. Training data consists of base-model rollouts on the Reddit conversations corpus (for length control and HH experiments) or the TL;DR dataset (for summarization). The paper does not report specific learning rates or batch sizes for CD-FUDGE training in the main text. **CD-Q training.** Trained via SGD on the objective in Equation 5. The target $v_t$ uses a stop gradient over $V_\theta$ in the next-state expectation (standard DQN-style stabilization). The paper does not report specific hyperparameters but notes that the simplest form of TD learning was used and that improvements from the deep RL literature (Hessel et al., 2018) are left to future work. **Online DPO and IPO baselines.** For fair comparison, the paper uses online versions of DPO and IPO. The policy is rolled out to generate two responses per prompt, the responses are scored by the reward model, and the DPO/IPO objective is optimized on these pairwise comparisons using the explicit reward values. This makes DPO/IPO more comparable to CD and PPO because they all use the same reward signal rather than fixed preference data. Several training runs with different regularizer hyperparameters and learning rates were performed to sweep a range of KL divergence values. **PPO baseline.** KL-regularized PPO is trained as in Ouyang et al. (2022), with the KL regularizer strength swept to achieve different points on the reward-KL tradeoff curve. **Evaluation on HH and summarization.** For measuring win rates, the paper uses PaLM 2-L (Unicorn) (Anil et al., 2023) as a zero-shot evaluator. The evaluator is given the dialogue context and two responses (one from the aligned policy, one from the base policy) and asked to select which is more helpful/harmless (or which summary is better). The detailed prompts used for zero-shot evaluation are provided in Appendix A. This is an LLM-as-judge evaluation, not a human evaluation—the paper acknowledges this implicitly by noting the reward model's own accuracy (~0.7 on test set, Table 1) as a separate metric. **Response length reward.** Defined as $r_{\text{length}}([x, y_T]) = \log(T / T_{\text{max}})$ where $T_{\text{max}} = 1024$. This is a simple, deterministic reward with no noise, chosen for the first experiment to establish baseline behavior before moving to noisy learned rewards. **Block size $M$.** The paper explores block sizes up to 32 tokens (Figure 9). Larger block sizes generally give better win-rate vs. KL tradeoffs because they allow the prefix scorer to evaluate more complete thoughts before making a selection. However, block sizes larger than 32 are not explored because "the efficiency gains against best-of-K would evaporate"—if $M$ becomes comparable to full sequence length, blockwise CD reduces to standard best-of-K. ## 4. Key Insights and Innovations ### Innovation 1: Reframing Inference-Time Control as Solving a Specific RL Objective (Not Heuristic Logit Tweaking) The paper's most fundamental intellectual contribution is not the prefix scorer architecture itself—which builds directly on FUDGE (Yang & Klein, 2021)—but rather **proving that a prefix scorer trained to estimate the value function $V^\star$ of the base policy enables sampling from the exact optimal policy of a tokenwise KL-regularized RL objective**. This converts controlled decoding from a heuristic practice ("train a classifier on partial sequences and use it to nudge logits") into a principled optimization procedure with formal guarantees. **What the field did before.** Prior controlled generation methods—FUDGE (Yang & Klein, 2021), GeDi (Krause et al., 2021), DIRECTOR (Arora et al., 2022), NADO (Meng et al., 2022)—all operated on the same general principle: train an auxiliary model to score partial sequences according to some desired attribute, then adjust the base model's token probabilities during generation. But these methods were developed within the NLP community without explicit connections to reinforcement learning. FUDGE, for example, was derived from Bayes' rule: $p(z \mid \text{attribute}) \propto p(z) p(\text{attribute} \mid z)$, where the prefix scorer estimates $p(\text{attribute} \mid \text{prefix})$. This is intuitive but left fundamental questions unanswered: what optimization problem is being solved? Under what conditions is the reweighting optimal? How should the strength of the prefix scorer's influence be set relative to the base model's probabilities? The field's dominant framing for alignment was training-time RL (PPO, DPO, IPO), where the optimization objective is explicit—maximize expected reward subject to a KL penalty—but the solution modifies the generator's weights. Inference-time methods were seen as convenient approximations at best, with no clear relationship to the RL objective that training-time methods optimize. **What CD does differently.** The paper's Theorem 2.1 changes the game: it shows that the tokenwise KL-regularized RL objective has a closed-form optimal policy that depends only on the base model $\pi_{\text{ref}}$ and the value function $V^\star$. This is not an approximation or a heuristic—it is the *exact* solution. The implication is profound: **if you can learn $V^\star$, you can implement the optimal RL policy without ever updating the base model's weights**. Training-time methods (PPO, DPO) and inference-time methods (CD) are therefore solving the *same* optimization problem through different computational mechanisms—one by modifying the generator, the other by learning a value function and using it during decoding. This reframing matters because it provides what was previously missing: **a theoretical equivalence between training-time and inference-time alignment**. It means CD is not "cheating" by avoiding model updates; it is an alternative computational path to the same mathematical optimum. The practical advantages of CD (modularity, transfer, multi-objective combination) are therefore not paid for by sacrificing optimality—they come from rearranging *how* the optimization is implemented, not *what* is optimized. **The significance of the tokenwise vs. sequence-level distinction.** The paper is careful to note that its tokenwise RL objective is "more restrictive than the sequence-level RL used to design RLHF and DPO" (Remark in Section 2). This is not a weakness—it is what enables the closed-form solution. Sequence-level KL-regularized RL does not generally admit a simple per-token reweighting; the optimal policy depends on the full trajectory distribution in complex ways. By formulating the problem tokenwise, the paper makes it analytically tractable. The empirical finding that blockwise CD (which approximates sequence-level optimization) outperforms tokenwise CD suggests that the tokenwise objective may be *too* restrictive, but this does not diminish the theoretical contribution: the paper shows that a clean optimization story exists for a well-defined objective, and that deviations from that objective (via blockwise selection) can improve empirical performance while retaining a principled foundation. **Tie to evidence.** The proof of Theorem 2.1 (Appendix C) shows that the objective $J_\lambda$ can be rewritten as negative KL divergence between $\pi$ and a target distribution $q_\lambda$ plus a constant, making optimality immediate. The CD-FUDGE convergence result (Theorem 3.1, Theorem C.2) demonstrates that training on base-model rollouts with MSE loss produces prefix scorers that converge to $V^\star$—connecting the practical training procedure back to the theoretical optimum. These are not hand-wavy claims; they are formal statements with assumptions and convergence conditions. --- ### Innovation 2: Proving FUDGE Optimizes a Well-Defined RL Objective (Retroactive Theoretical Foundation) A substantial secondary contribution is providing a **retroactive theoretical justification for FUDGE**, a method published three years prior that the community had treated as an empirically useful but theoretically unmotivated heuristic. The paper's Theorem 3.1 shows that FUDGE—when trained on data generated by rolling out the base model—performs stochastic gradient descent on a loss function whose gradient is an unbiased estimator of the gradient of the true value function objective $\mathcal{L}^\star$. In other words, **FUDGE was implicitly doing policy evaluation for the base policy all along, converging to the value function that enables optimal KL-regularized RL**. **What the field thought before.** FUDGE (Yang & Klein, 2021) was introduced as a method for controlled text generation using "future discriminators"—classifiers trained to predict, from a partial sequence, whether the final output would satisfy some attribute (e.g., positive sentiment, formal register). The derivation was Bayesian: factor the desired conditional distribution $p(\text{text} \mid \text{attribute})$ using a learned $p(\text{attribute} \mid \text{partial text})$ term. The connection to RL, value functions, or optimal control was never made. FUDGE was understood as an inference-time conditioning trick, not as a solution to an optimization problem. The consequence was that FUDGE existed in a separate conceptual universe from training-time alignment methods. There was no reason to expect FUDGE and PPO to be optimizing the same thing, no way to compare their theoretical properties, and no framework for understanding when FUDGE would succeed or fail. The method spread through empirical demonstration rather than theoretical understanding. **What CD contributes to this understanding.** Lemma C.1 proves that $\mathbb{E}_{y \sim \pi_{\text{ref}}}[\nabla_\theta \mathcal{L}_F(\theta)] = \nabla_\theta \mathcal{L}^\star(\theta)$, where $\mathcal{L}_F$ is the FUDGE loss (MSE between prefix scorer prediction and final reward) and $\mathcal{L}^\star$ is the ideal value function loss (MSE between prefix scorer prediction and true $V^\star$). The proof works by expanding the squared error, applying the law of total expectation to replace the noisy final reward with $V^\star$ in cross-terms, and observing that variance terms involving $r^2$ are constant with respect to $\theta$. **Why this is more than a technical curiosity.** This result transforms FUDGE from a heuristic into a method with known optimization guarantees. It explains *why* FUDGE works: because it is approximately solving the right problem. It also explains *when* FUDGE might fail: if training data does not come from the base model (violating the on-policy assumption), the gradient estimator is biased, and convergence to $V^\star$ is not guaranteed. And it suggests *how* FUDGE might be improved: by adopting techniques from the value function learning literature (better loss functions, temporal difference targets rather than Monte Carlo returns), which is exactly what CD-Q does. **The relationship between CD-FUDGE and CD-Q as complementary training strategies.** The paper's presentation of two training methods—one on-policy with Monte Carlo targets (CD-FUDGE), one off-policy with TD targets (CD-Q)—creates a useful taxonomy. CD-FUDGE is simpler to implement (just roll out the base model and regress on final rewards) but produces noisier value estimates (evidenced by Figure 13, where CD-FUDGE length predictions are substantially more scattered than CD-Q's). CD-Q requires more engineering (Bellman backups, stop gradients, access to $\pi_{\text{ref}}$ during training) but produces cleaner value estimates that enable superior blockwise performance. This is not presented as a "CD-Q beats CD-FUDGE" story—both have the same theoretical guarantee of converging to $V^\star$. Rather, it is an observation that **the quality of value estimates matters enormously for downstream use**, and that TD learning provides a practical advantage for the blockwise selection mechanism that the paper finds most effective. --- ### Innovation 3: Blockwise Control as a Practical Bridge Between Tokenwise RL and Best-of-K The paper identifies and resolves a tension that is visible throughout the alignment literature but rarely articulated clearly: **tokenwise RL methods (like tokenwise CD or PPO) theoretically optimize the right objective, but empirically underperform sequence-level selection methods (like best-of-K) on reward-KL tradeoffs**. Blockwise CD is the paper's proposed resolution—it retains best-of-K's excellent empirical properties while addressing its operational weaknesses (latency, sample inefficiency) and maintaining a connection to the RL framework. **The tension.** This is not a new observation—the paper explicitly cites Gao et al. (2023, Figure 1) and Rafailov et al. (2023, Figure 3), both of which showed that best-of-K achieves better reward-KL tradeoffs than KL-regularized PPO. Yang et al. (2024) later provided theoretical reasoning: best-of-K is an almost optimal solution to the KL-regularized RL problem. The implication is uncomfortable for the field: the methods that directly optimize the RL objective (PPO, tokenwise CD) are being outperformed by a simple rejection sampling procedure that makes no attempt to solve the optimization—it just generates a lot and picks the best. **What blockwise CD contributes to understanding this.** Rather than accepting best-of-K as an empirically superior but operationally flawed approach, the paper asks: *what property of best-of-K makes it work so well, and can we isolate that property in a more practical mechanism?* The answer, implicit in the paper's design, is that **evaluating complete multi-token trajectories (rather than individual token choices) provides a more reliable signal for selection**. Tokenwise methods must commit to individual token decisions based on value estimates that may be noisy or myopic. Best-of-K evaluates whole sequences after the fact, integrating over all token-level decisions simultaneously. Blockwise CD splits the difference: it evaluates trajectories of length $M$ (long enough to capture meaningful semantic content, short enough to enable early intervention) and selects among them using the prefix scorer. **The innovation is in identifying $M$ as the critical parameter controlling the tokenwise-to-best-of-K spectrum.** When $M = 1$, blockwise CD reduces to a greedy tokenwise selection—picking the single best next token at each step. When $M$ equals the full sequence length, blockwise CD reduces to standard best-of-K—one selection at the end of the full response. By varying $M$, the method interpolates between these extremes, and the empirical results (Figure 9) show that larger $M$ consistently improves reward-KL tradeoffs, confirming that the benefits of best-of-K derive from evaluating longer horizons. **Operational significance beyond the RL connection.** Even if one does not care about the theoretical link to KL-regularized RL, blockwise CD offers a concrete practical improvement over best-of-K: it reduces latency from full-sequence-length to $M$ tokens (enabling streaming applications), and it reduces the required $K$ by up to ~8× for equivalent performance (Experiment 1, where blockwise CD-Q with $K=6$ matches best-of-K with $K=50$). This is not a theoretical claim—it is a direct consequence of making more frequent selection decisions. Each blockwise selection gives the system a chance to correct course; best-of-K only gets one chance at the very end, so it needs many more candidates to achieve the same probability of finding a good one. --- ### Innovation 4: Demonstrating That the Value Function—Not the Generator—Is the Transferable Component for Alignment The paper includes an experiment (Experiment 5, Figures 7 and 8) showing that a prefix scorer trained on one base model (PaLM 2-XXS) transfers to different base models (PaLM 2-S and PaLM 2-XS) with no retraining and no performance degradation relative to best-of-K. This result, while presented modestly, encodes a **non-obvious claim about the nature of alignment**: the value function may be substantially less model-specific than the aligned generator. **What the field would have expected.** The default assumption in alignment research is that the aligned policy is tightly coupled to the base model. PPO fine-tunes the generator; DPO optimizes the generator directly from preferences. If you swap the base model, you need to retrain—this is obvious and universally accepted. A prefix scorer trained to estimate $V^\star$ for a specific $\pi_{\text{ref}}$ would, under this default assumption, be expected to transfer poorly because $V^\star$ depends on $\pi_{\text{ref}}$ by definition: it is the expected reward when completing generations using $\pi_{\text{ref}}$, not some universal quality measure of partial sequences. **What the transfer result implies.** The fact that transfer works—and works *well*, matching best-of-K performance—suggests that the value function $V^\star$ for different members of the PaLM 2 family is similar enough that a scorer trained on one model serves as an effective proxy for another. This could be because the models produce similar completions from similar prefixes (their output distributions are close), or because the value function captures something about the *semantic quality* of partial sequences that is relatively invariant to which specific language model produced them. The paper does not disentangle these explanations, but either one challenges the assumption that alignment must be model-specific. **Practical significance.** If prefix scorers transfer across base model versions, CD enables a deployment pattern where the prefix scorer is trained once and reused across base model updates—a substantial cost saving over retraining PPO or DPO policies for each new checkpoint. The paper's Experiment 7 (Figure 10) takes this further: CD-Q is applied on top of a DPO-fine-tuned model without retraining the prefix scorer, and the combination achieves the best overall win-rate vs. KL tradeoff curve. This suggests that CD can be layered on top of training-time alignment as a complementary mechanism—the DPO update shifts the base distribution toward higher-reward regions, and CD-Q provides additional inference-time steering, with no coordination between the two training procedures. **The limits of this claim.** The paper only demonstrates transfer within the PaLM 2 family. Transfer across dramatically different architectures, model scales, or training paradigms may fail. The paper does not claim universality, and a systematic study of when value functions transfer is left to future work. But even the limited demonstration within a model family is practically valuable: in production, base model updates within the same family are common, and the finding that prefix scorers survive these updates without retraining makes CD's modularity argument substantially stronger. --- ### Innovation 5: Verifier Over-Optimization as a First-Class Phenomenon in Test-Time Scaling While reward hacking / over-optimization is well-documented in the RLHF literature, this paper provides some of the first clear evidence that **the same phenomenon governs test-time search scaling** and is the primary bottleneck preventing unbounded improvements from additional compute. The evidence is concrete: beam search degrades easy-problem performance at high budgets (Figure 3, right); lookahead search—the most powerful optimizer—paradoxically performs *worst* overall (Figure 3, left); and qualitative examples in Appendix M show search producing degenerate outputs (repetitive low-information steps, overly short solutions) that score highly under the PRM. This finding is significant because it shifts the narrative around test-time compute from "more is better" to "more is better only up to the verifier's reliability frontier." It explains why prior work found negative results for sophisticated search methods: those studies likely pushed past the over-optimization threshold. It also implies that **improving verifier robustness is the key bottleneck** for further scaling test-time compute, not improving search algorithms. The paper's compute-optimal policy can be understood partly as a way to stay *below* the over-optimization threshold per difficulty level—using weaker optimization (best-of-N) where the verifier is reliable (easy problems) and stronger optimization (beam search) only where the verifier signal has more room to provide genuine guidance (medium problems). **The practical implication of this finding.** The paper identifies a concrete engineering bottleneck: improving verifier robustness—through better training data, adversarial regularization, or ensemble methods—would directly raise the ceiling on test-time compute scaling, whereas developing more sophisticated search algorithms (the natural first instinct) may be counterproductive if they over-optimize the existing verifier more aggressively. ## 5. Experimental Analysis ### Evaluation Methodology - **Dataset.** The paper uses three datasets across its experiments: **(1)** the DSTC8 Reddit conversations corpus (Microsoft, 2019) for training and evaluating response length control—this contains millions of multi-turn conversations from Reddit threads; **(2)** the Anthropic HH (helpfulness and harmlessness) dataset (Bai et al., 2022) for training reward models and prefix scorers on human preference data—this is a benchmark where the assistant completes the next turn in a conversation with a human, with pairwise preferences provided; **(3)** the TL;DR summarization dataset (Stiennon et al., 2020) for training a summarization quality reward model—this contains Reddit posts with human preference annotations between pairs of summarization candidates. The paper uses standard train/test splits for each dataset, though specific sizes of evaluation sets are not reported in the main text except for the HH preference accuracy evaluation which uses 1500 side-by-side comparisons (Table 1). - **Base model(s).** All primary experiments use PaLM 2-XXS (Gecko) (Anil et al., 2023) as the base generative model. The prefix scorer is also fine-tuned from PaLM 2-XXS—sharing the same architecture and initial weights. The authors state that this model is chosen because CD is "motivated by tradeoffs between throughput, latency, and performance" (Section 7), and XXS represents a small, deployment-friendly scale where inference-time overhead is practically relevant. For the base-model transfer experiments (Experiment 5), the prefix scorer trained on PaLM 2-XXS is applied without retraining to **PaLM 2-S (Bison)** for length control and **PaLM 2-XS (Otter)** for HH control (Figures 7 and 8). For zero-shot evaluation of HH and summarization quality, the paper uses **PaLM 2-L (Unicorn)** (Anil et al., 2023) as an LLM judge, prompted to rank two responses side-by-side (prompts in Appendix A). - **Metrics.** The paper reports three primary metrics, all plotted against KL divergence between the aligned policy and the base policy: - **Normalized expected reward**: $\mathbb{E}_{x \sim \mu} \mathbb{E}_{y \sim \pi_\theta(\cdot|x)} r(x, y)$, normalized to the expected reward of the reference policy $\pi_{\text{ref}}$. For length control, this is the average log-length of generated responses divided by the baseline's average log-length (Figure 3, y-axis labeled "Average length (normalized)"). For HH and summarization, the raw reward is the log-probability from the trained Bradley-Terry reward model. - **Win rate against base policy**: $\mathbb{E}_{x \sim \mu} \mathbb{E}_{y \sim \pi_\theta(\cdot|x)} \mathbb{E}_{z \sim \pi_{\text{ref}}(\cdot|x)} \mathbf{1}[r(x, y) > r(x, z)]$, i.e., the fraction of prompts where the aligned policy's response scores higher than the base policy's response according to the reward model (for length) or an LLM judge (for HH and summarization). This is the primary metric in Figures 4, 5, 6, 8, 9, 10, 12, and 14. - **KL divergence**: the sequence-level KL divergence $\text{KL}(\pi \| \pi_{\text{ref}}) = \mathbb{E}_{x \sim \mu} \mathbb{E}_{y \sim \pi(\cdot|x)} \{\log \pi(y|x) - \log \pi_{\text{ref}}(y|x)\}$, used as the x-axis in all tradeoff plots to measure deviation from the base model. For best-of-K, the paper uses the upper bound $\log(K) - (K-1)/K$ (Stiennon et al., 2020; Beirami et al., 2024). For blockwise CD, it uses the extended bound $\mathbb{E}_{x \sim \mu} (\log(K) - (K-1)/K) \lceil L_x / M \rceil$, where $L_x$ is the number of decoded tokens and $M$ is the block size. For tokenwise methods, the KL is estimated from the explicitly defined per-token policies. The paper focuses on KL values smaller than 10 to avoid the overfitting regime (Eisenstein et al., 2023). Additionally, Table 1 reports **classification accuracy** of the prefix scorers and reward model on the Anthropic HH preference prediction task, treating each as a binary classifier for which of two responses is preferred. - **Baselines.** The paper compares against eight baselines spanning both inference-time and training-time alignment paradigms: - **CD-FUDGE** (Yang & Klein, 2021): tokenwise control using a prefix scorer trained on base-model rollouts with Monte Carlo regression targets (Equation 4). Also evaluated in a blockwise variant that the paper introduces (blockwise CD-FUDGE), extending the blockwise idea to the FUDGE-trained scorer. - **KL-regularized PPO** (Ouyang et al., 2022): a training-time method that fine-tunes the base model using proximal policy optimization with a KL penalty against the reference policy. Multiple checkpoints with different KL regularizer strengths are trained to sweep the tradeoff curve. - **DPO** (Rafailov et al., 2023): a training-time method that optimizes the generator directly from pairwise preference data. The paper uses online DPO, where the policy is rolled out to generate two responses per prompt, scored by the reward model, and the DPO objective is optimized on these self-generated comparisons. - **IPO** (Azar et al., 2023): similar to DPO but with a modified objective designed to avoid DPO's degeneration issues. Also used in an online variant with self-generated pairwise data. - **Best-of-K** (Nakano et al., 2021; Stiennon et al., 2020): the standard inference-time method where K complete responses are sampled from the base model, ranked by the reward model, and the highest-scoring response is returned. K is swept to vary the KL bound. - **Blockwise CD-FUDGE**: the paper's proposed extension of FUDGE to the blockwise best-of-K mechanism, using the CD-FUDGE-trained prefix scorer for block ranking. - **DPO + CD-Q (blockwise)**: a combined approach where CD-Q's blockwise mechanism is applied on top of a DPO-fine-tuned base model (Experiment 7, Figure 10). - **DPO + Best-of-K**: DPO with standard best-of-K sampling at inference time, used as a throughput-matched comparison in Experiment 8 (Figures 11 and 12). - **Generation budget / compute accounting.** For tokenwise CD, there is no explicit K parameter—the cost is one prefix scorer call per generated token (per candidate evaluated at each step). For blockwise CD, the budget is parameterized by K (number of candidate blocks) and M (block length in tokens). The paper fixes K and varies M (or vice versa) to sweep different points on the reward-KL tradeoff curve. For best-of-K, the budget is simply K—the number of complete sequences generated and scored. For PPO, DPO, and IPO, there is no inference-time sampling budget (they generate one response), but different training runs with varying regularizer strengths produce policies at different KL divergences from the base model. In Experiment 8 (Figure 11), a fixed inference throughput budget is imposed by setting K for blockwise CD-Q and K for best-of-K applied on top of DPO to the same value, so both methods generate the same number of tokens per prompt (one response from CD-Q via K parallel decodes per block, K responses from DPO for best-of-K ranking). - **Cross-validation / statistical protocol.** The paper does not report cross-validation or statistical significance testing. Results are presented as point estimates in tradeoff plots (reward/KL curves) without error bars or confidence intervals. For the HH preference accuracy evaluation (Table 1), separate training and test accuracy numbers are reported on 1500 examples, but no standard deviations or significance tests are provided. For DPO, IPO, and PPO, multiple training runs were performed "varying regularizer hyperparameters and learning rates to reach comparable KL against other methods" (Section 4.5), but the selection criterion for reporting a particular run is not specified. The LLM-as-judge evaluation (using PaLM 2-L for HH and summarization win rates) is deterministic given the prompt and responses, but no calibration of the judge against human preferences is reported. ### Main Quantitative Results #### Experiment 1: Response Length Control **Headline result.** Blockwise CD-Q achieves the best length-vs-KL tradeoff among all methods, matching best-of-K's performance while requiring up to **~8× fewer samples** (K=6 vs. K=50 for equivalent length and KL), and substantially outperforming all training-time baselines (PPO, DPO, IPO). Figure 3 reports the core comparison. At a normalized average length of approximately 1.5 (50% longer than the base model's average response), the methods achieve the following approximate KL divergences (read from the figure): - Blockwise CD-Q (K=6): KL ≈ 3 - Best-of-K (K=50): KL ≈ 3 - CD-Q (tokenwise): KL ≈ 5 - CD-FUDGE (tokenwise): KL ≈ 5 - CD-FUDGE (blockwise): KL ≈ 9 - Online DPO/IPO/PPO: KL ≈ 6–8 At a fixed KL budget of approximately 5, the normalized average lengths are approximately: - Blockwise CD-Q (K=6): ~1.65 - Best-of-K (K=50): ~1.68 (slightly higher, but requires ~8× more samples) - CD-Q (tokenwise): ~1.50 - CD-FUDGE (tokenwise): ~1.55 - Online DPO/IPO: ~1.35 (tokenwise CD methods clearly dominate these) - Blockwise CD-FUDGE: ~1.15 (substantially worse, barely above baseline) - PPO: ~1.25 The paper annotates the figure with explicit comparison: "K=6" for blockwise CD-Q versus "K=50" for best-of-K, highlighting the sample efficiency gap. Tokenwise CD-Q and CD-FUDGE outperform DPO, IPO, and PPO across the KL range, but are themselves dominated by blockwise CD-Q and best-of-K at moderate-to-high KL values. **Key insight from this experiment.** Best-of-K achieves a better reward-KL tradeoff than KL-regularized PPO, consistent with prior findings (Gao et al., 2023; Rafailov et al., 2023). Blockwise CD-Q matches best-of-K's performance while being more sample-efficient, and substantially outperforms blockwise CD-FUDGE. The paper attributes this gap to the noisy value estimates produced by CD-FUDGE, documented in Figure 13 (Appendix B), which make the blockwise ranking unreliable. #### Experiment 2: Helpfulness and Harmlessness (HH) Control **Headline result.** Blockwise CD-Q substantially outperforms all training-time baselines (IPO, PPO) on HH win rate vs. KL tradeoff, but does not match best-of-K. Tokenwise controllers (CD-Q and CD-FUDGE) offer minimal improvement over the base policy. Figure 4 reports the HH win rate (as judged by PaLM 2-L zero-shot, y-axis) against KL divergence (x-axis). At a KL of approximately 2.5: - Best-of-K: win rate ≈ 0.73 (highest) - Blockwise CD-Q: win rate ≈ 0.68 - Blockwise CD-FUDGE: win rate ≈ 0.66 - IPO: win rate ≈ 0.55 (barely above the base policy) - PPO: win rate ≈ 0.55 - Tokenwise CD-Q: win rate ≈ 0.53 - Tokenwise CD-FUDGE: win rate ≈ 0.52 At KL ≈ 5, blockwise CD-Q reaches a win rate of approximately 0.72, still trailing best-of-K's 0.78–0.80 range. IPO and PPO remain below 0.60 across all KL values explored. **The blockwise advantage is qualitatively larger here than for length control.** Tokenwise CD methods barely improve over the base policy, while blockwise methods provide a substantial boost. The paper attributes this to the noisy, learned nature of the HH reward—the prefix scorer's per-token value estimates may be too noisy for effective tokenwise reweighting, whereas blockwise selection (which integrates over M tokens before making a decision) remains effective. **Table 1** provides a diagnostic: the Reward-XXS model achieves 0.804 training accuracy and 0.709 test accuracy on pairwise HH preference prediction. CD-Q and CD-FUDGE, when used as classifiers on the same task, achieve only ~0.63 train accuracy and ~0.63 test accuracy (CD-FUDGE: 0.632/0.629; CD-Q: 0.624/0.631). This gap in discriminative accuracy (~0.6 vs. ~0.7) indicates that the prefix scorers are weaker classifiers than the reward model itself, which the paper identifies as a likely reason neither blockwise CD-Q nor blockwise CD-FUDGE matches best-of-K (which uses the stronger Reward-XXS directly). The paper frames this as "an area for future investigation to improve the training using value function learning methods better suited to noisy reward environments." #### Experiment 3: Summarization Quality Control **Headline result.** Blockwise CD-Q outperforms IPO but falls short of best-of-K, mirroring the HH pattern. Figure 5 reports summarization quality win rate (PaLM 2-L judge) vs. KL divergence. At KL ≈ 2: - Best-of-K: win rate ≈ 0.68 - Blockwise CD-Q: win rate ≈ 0.63 - IPO: win rate ≈ 0.56 At KL ≈ 5: - Best-of-K: win rate ≈ 0.72 (approximately) - Blockwise CD-Q: win rate ≈ 0.67 (approximately) - IPO: win rate ≈ 0.58 (approximately) The paper only compares these three methods in this experiment (best-of-K, CD-Q blockwise, IPO), omitting PPO, DPO, CD-FUDGE, and tokenwise CD variants. The number of data points on the CD-Q curve is small (approximately 3–4), making the tradeoff curve less finely characterized than in Experiments 1 and 2. #### Experiment 4: Multi-Objective Control (HH + Length) **Headline result.** CD enables dynamic, inference-time adjustment of the tradeoff between helpfulness/harmlessness and response length using independently trained prefix scorers—a capability unavailable to training-time methods. The tradeoff is achieved by linearly combining the HH and length prefix scorer outputs with configurable weights. Figure 6 plots the joint (normalized average length, HH win rate) space. The results are reported for blockwise CD-FUDGE specifically (the paper only considers this variant for the multi-objective experiment). Key observations: - Applying the HH prefix scorer alone increases both HH win rate (from 0.50 baseline to approximately 0.62) AND response length (from normalized 1.0 to approximately 1.18). This is consistent with prior findings that optimizing for helpfulness tends to produce more verbose outputs. - Adding a negative weight on the length prefix scorer reduces the length increase: with a weight ratio that balances the two objectives, the length returns to approximately 1.0 (baseline level) while the HH win rate drops to approximately 0.58. - Different linear combinations trace out a tradeoff curve in the (length, HH win rate) plane, with the extremes being (low HH gain, no length increase) and (higher HH gain, substantial length increase). The paper does not report the specific weight values ($\alpha_i$) used for each point in Figure 6, nor does it evaluate whether the same tradeoff curve could be achieved by training separate models for each objective combination (the counterfactual that would be required for training-time methods). The claim is that this *could not be done at all* with PPO/DPO/IPO without retraining—which is definitionally true since those methods bake the reward into the generator weights—but the paper does not compare against a retrained multi-objective PPO/DPO baseline to quantify whether CD's combined performance is competitive with what retraining would achieve. #### Experiment 5: Prefix Scorer Transfer to Unseen Base Models **Headline result.** Blockwise CD-Q's prefix scorer, trained on PaLM 2-XXS, transfers to PaLM 2-S (for length control) and PaLM 2-XS (for HH control) without retraining, performing on par with best-of-K on the new base models. **Length control transfer (Figure 7).** The prefix scorer trained with CD-Q on PaLM 2-XXS is applied to PaLM 2-S. The normalized average length vs. KL curve for blockwise CD-Q on PaLM 2-S is plotted alongside best-of-K applied directly to PaLM 2-S. The two curves are essentially overlapping across the full KL range (approximately KL 0.5–8). For example, at normalized length ~1.4, blockwise CD-Q achieves KL ≈ 4, matching best-of-K at the same KL. At KL ≈ 6, blockwise CD-Q achieves normalized length ~1.65, matching best-of-K again. **HH control transfer (Figure 8).** The prefix scorer trained on PaLM 2-XXS is applied to PaLM 2-XS. The HH win rate vs. KL curve for blockwise CD-Q on PaLM 2-XS is plotted alongside best-of-K. Again, the two curves track each other closely. At KL ≈ 2, both achieve win rate ≈ 0.66; at KL ≈ 5, both achieve win rate ≈ 0.70. The paper notes that "PPO/DPO/IPO could not be used without re-training in this experiment," underscoring CD's modularity advantage. #### Experiment 6: Impact of Block Size M on Blockwise CD **Headline result.** Larger block sizes consistently improve the HH win rate vs. KL tradeoff, with M=32 performing best among the tested values. Figure 9 reports HH win rate vs. KL for blockwise CD-Q with block sizes M ∈ {4, 8, 16, 32}. The curves are ordered: at any fixed KL, larger M yields higher win rate. For example, at KL ≈ 3: - M = 32: win rate ≈ 0.70 - M = 16: win rate ≈ 0.68 - M = 8: win rate ≈ 0.65 - M = 4: win rate ≈ 0.62 The gap between M=32 and M=16 is smaller than the gap between M=8 and M=4, suggesting diminishing returns as M increases. The paper does not explore block sizes larger than 32, noting that "the efficiency gains against best-of-K would evaporate"—if M approaches the full sequence length, blockwise CD reduces to standard best-of-K, losing the latency and streaming advantages. #### Experiment 7: CD-Q Applied on Top of a DPO Base Model **Headline result.** Applying blockwise CD-Q on top of a DPO-fine-tuned model (without retraining the prefix scorer) yields the overall best win-rate vs. KL tradeoff curve, narrowly outperforming vanilla blockwise CD-Q at equivalent KL, and achieving similar performance with smaller K (K=8 vs. K=32 for KL ≈ 5). Figure 10 plots HH win rate vs. KL for three approaches: vanilla blockwise CD-Q, DPO alone (multiple checkpoints at different KL values), and "DPO + CD-Q (blockwise)." The DPO + CD-Q combination is constructed by taking a DPO checkpoint with a certain KL divergence from the base model (e.g., 2.5) and then applying blockwise CD-Q with K chosen to add additional KL (e.g., K=8 adds approximately 2.5 KL, for a total of ~5). Key comparisons (annotated on the figure): - At KL ≈ 5, "DPO + CD-Q (blockwise)" with K=8 achieves win rate ≈ 0.72, while "CD-Q (blockwise)" alone with K=32 achieves win rate ≈ 0.71. The combined variant achieves slightly better performance with **4× smaller K**. - At lower KL (~2–3), the combined variant and vanilla blockwise CD-Q are approximately tied. - DPO alone achieves substantially lower win rates at any given KL, e.g., at KL ≈ 3, DPO achieves win rate ≈ 0.58 vs. CD-Q's 0.69. The paper notes that this experiment demonstrates CD's modularity: the prefix scorer can be trained once and then applied to any base model (including DPO-fine-tuned ones) without coordination between training procedures. The combined variant presumably benefits from the DPO shift moving the base distribution closer to high-reward regions, reducing the work the prefix scorer must do during blockwise selection. #### Experiment 8: Fixed Inference Throughput Budget Comparison **Headline result.** When matched on inference throughput (same K, same number of generated tokens per response), blockwise CD-Q consistently outperforms "DPO + best-of-K," with the gap widening at larger K. **Length control (Figure 11).** The experiment fixes K ∈ {4, 8, 16}. For each K, blockwise CD-Q's tradeoff curve is obtained by varying M (the block size), while "DPO + best-of-K" uses standard best-of-K sampling on top of the DPO policy. At K=4: - Blockwise CD-Q: at normalized length ~1.35, KL ≈ 1.5 - DPO + best-of-K: at normalized length ~1.30, KL ≈ 1.5 At K=8, the gap widens: at KL ≈ 2.5, blockwise CD-Q achieves normalized length ~1.55 vs. DPO + best-of-K at ~1.35. At K=16, the gap widens further: at KL ≈ 4, blockwise CD-Q achieves ~1.70 vs. DPO + best-of-K at ~1.45. **HH control (Figure 12).** A similar comparison at K=4 for HH: "DPO + CD-Q (blockwise)" is compared against "DPO + Best-of-K." At K=4, the two methods are approximately on par, both achieving win rate ≈ 0.68 at KL ≈ 2–3. The paper notes this equivalence, suggesting that for small K, the advantages of blockwise CD may be less pronounced when combined with DPO. **Additional result (Figure 14, Appendix B).** The paper extends Experiment 7 by applying blockwise CD-Q to four different DPO checkpoints covering a range of KL divergences (0.3, 1.15, 2.25, 3.25). The combined approaches achieve win rates up to ~0.72 at KL values between 3 and 5. The figure annotates that points at win rate 0.7 can be achieved with K=4 for the combined approach whereas vanilla blockwise CD-Q requires K=32—an 8× reduction in sample complexity. ### Ablation Studies and Robustness Checks - **CD-FUDGE vs. CD-Q as value predictors (Figure 13, Appendix B).** When used to predict the length of fully decoded responses on the Reddit test set, CD-Q's predictions closely track the actual lengths, while CD-FUDGE's predictions are substantially noisier. The test examples are ordered by actual response length along the x-axis; CD-Q's predicted lengths follow the monotonic trend cleanly, while CD-FUDGE's predictions show large variance and frequent over/under-estimation. This ablation explains the performance gap in blockwise sampling: accurate relative ranking of candidate blocks requires reliable value estimates, and CD-FUDGE's noise degrades ranking quality. - **Block size sweep for HH (Figure 9).** Larger M consistently improves win-rate vs. KL tradeoffs, with M=32 being the best among tested values. The paper does not report computational cost scaling with M, nor does it characterize the latency implications (larger M means longer blocks must be decoded before selection can occur, partially eroding the streaming advantage over best-of-K). The decision not to explore M > 32 is justified by the concern that efficiency gains over best-of-K would "evaporate," but no quantitative analysis of this tradeoff is provided. - **DPO checkpoint selection for combined approach (Figure 14, Appendix B).** Applying blockwise CD-Q to four different DPO checkpoints (spanning KL values from 0.3 to 3.25) shows that the combined approach works across a range of base model KL divergences. The highest win rate (~0.72) is achieved with a DPO checkpoint at KL ≈ 2.25 combined with blockwise CD-Q at K=4. At very low DPO KL (0.3), the combined performance is close to vanilla CD-Q; at high DPO KL (3.25) with moderate CD-Q K (8), the performance is similar to the optimal configuration. This suggests robustness to the specific DPO checkpoint chosen, though the paper does not systematically sweep all combinations of DPO KL and CD-Q K to identify the Pareto frontier. - **Reward model vs. prefix scorer classification accuracy (Table 1).** The Reward-XXS model achieves 0.709 test accuracy on HH preference prediction, while CD-Q and CD-FUDGE achieve only ~0.63. This gap quantifies the information loss from compressing the reward model into a prefix scorer trained via value function regression versus using the reward model directly (as best-of-K does). The paper presents this as a diagnostic rather than an ablation, but it effectively ablates the key limitation of CD: the prefix scorer is a weaker discriminator than the reward model it approximates, which caps CD's performance relative to best-of-K. - **Tokenwise vs. blockwise for different reward types.** Across Experiments 1–3, the relative advantage of blockwise over tokenwise CD varies by reward type: for length (a clean, deterministic reward), tokenwise CD-Q and CD-FUDGE perform respectably (achieving ~1.5–1.6 normalized length at KL=5, Figure 3); for HH (a noisy, learned reward), tokenwise methods barely improve over the base policy (win rate ~0.53 at KL=2, Figure 4), while blockwise methods provide substantial gains. The paper does not systematically ablate why noisy rewards disproportionately harm tokenwise control, but the implied mechanism is that per-token value estimates become unreliable under reward noise, whereas block-level aggregation averages out some of this noise. - **Negative result: CD-FUDGE fails at blockwise control.** Blockwise CD-FUDGE substantially underperforms blockwise CD-Q (Figures 3 and 4) and even tokenwise CD-FUDGE in some regimes. This is attributed to the noise in CD-FUDGE's value estimates (Figure 13) and represents a non-obvious finding: the training procedure that works adequately for tokenwise reweighting (where soft probability adjustments are forgiving of value noise) fails for hard argmax selection (where ranking errors are catastrophic). The paper does not explore whether this could be remedied by training CD-FUDGE with more rollouts (reducing Monte Carlo variance in the targets) or using a different loss function. - **Multi-objective combination (Figure 6).** Only blockwise CD-FUDGE is evaluated for multi-objective control; CD-Q's multi-objective capability is not demonstrated. The paper does not ablate whether the choice of prefix scorer training method matters for the fidelity of objective combination (e.g., whether CD-Q's cleaner value estimates would produce better-calibrated tradeoffs). ### Critical Assessment **Claim 1 from the executive summary: CD provably samples from a solution to the KL-regularized RL objective.** The theoretical results (Theorem 2.1, Theorem 3.1) establish that CD, when using the *true* value function $V^\star$, implements the exact optimal policy, and that CD-FUDGE training on base-model rollouts converges to $V^\star$ under regularity assumptions. However, the experimental validation of this claim is incomplete: no experiment measures how close the learned $V_\theta$ is to $V^\star$ (beyond the diagnostic in Figure 13 for length prediction). The paper reports downstream metrics (win rate, normalized reward) but not value function error. This matters because the claim "provably samples from a solution" is about the procedure's asymptotic correctness, while the experiments demonstrate finite-sample empirical performance. The gap between the two is especially relevant given that CD's empirical performance varies substantially by method (CD-Q vs. CD-FUDGE, tokenwise vs. blockwise), suggesting that value function approximation error dominates in practice. The paper would be strengthened by a direct evaluation of value prediction accuracy (e.g., MSE between $V_\theta$ and Monte Carlo estimates of $V^\star$) across training steps and methods. **Claim 2: Blockwise CD matches best-of-K performance with ~8× fewer samples (K=6 vs. K=50 for length control).** This claim is supported by Figure 3 for the length control task only. The 8× figure specifically compares blockwise CD-Q with K=6 to best-of-K with K=50 at a normalized length of approximately 1.5 and KL ≈ 3. However, this comparison is not replicated for HH (Figure 4) or summarization (Figure 5), where blockwise CD-Q consistently *underperforms* best-of-K, making the sample-efficiency claim task-dependent. For HH, best-of-K reaches win rates that blockwise CD-Q never achieves at any KL budget, so the relevant question—"at a fixed win rate, how much smaller K does blockwise CD-Q need?"—is ill-posed within the explored K range. The paper does not explore whether increasing K for blockwise CD-Q beyond the values shown would eventually close the gap with best-of-K, or whether the performance ceiling is fundamental (due to the prefix scorer being a weaker discriminator than the reward model; Table 1). **Claim 3: CD transfers to unseen base models with no retraining and with no performance degradation.** The transfer experiments (Figures 7 and 8) support this claim *within the PaLM 2 family*. Blockwise CD-Q's prefix scorer, trained on PaLM 2-XXS, performs on par with best-of-K when applied to PaLM 2-S (length) and PaLM 2-XS (HH). However, several caveats limit the generality of this claim: - The models transferred between (XXS → S, XXS → XS) are architecturally similar (same PaLM 2 family) and differ in scale by roughly 1–2 orders of magnitude. Transfer across fundamentally different architectures or training paradigms (e.g., from PaLM to LLaMA) is not tested. - Performance is measured relative to best-of-K *on the new base model*, not relative to the performance CD-Q would achieve if retrained on the new base model. It is possible that retrained CD-Q would outperform the transferred version—the claim establishes a usable lower bound (transferred CD matches best-of-K) but not an upper bound (transferred CD might be leaving performance on the table). - Only blockwise CD-Q is evaluated for transfer; tokenwise CD transfer is not tested. **Claim 4: Multiple reward scorers can be combined at inference time with no additional training.** Demonstrated in Experiment 4 (Figure 6) for blockwise CD-FUDGE on HH + length. The demonstration is clear: different linear combinations of independently trained prefix scorers produce different (length, HH win rate) tradeoffs. However, the experiment has notable limitations: - Only two rewards are combined, and only in an additive linear fashion. More complex combinations (multiplicative, thresholded, min/max) are not explored. - The quality of the combined controller relative to a single model trained on the combined reward from scratch is not established. It is possible that joint training of a single prefix scorer on the weighted sum of rewards would outperform the post-hoc linear combination of independently trained scorers, because the former could learn interactions between objectives during training. - Only blockwise CD-FUDGE is tested; CD-Q's multi-objective performance is not evaluated. - The specific weight values for the linear combinations are not reported, making reproduction difficult. **Methodological weaknesses that affect interpretation of all results:** 1. **Single model family (PaLM 2).** All experiments use PaLM 2 variants as both the base generator and the architecture for the prefix scorer and reward models. The generality of CD to other model families (LLaMA, GPT, etc.) is untested. This is particularly relevant because CD's theoretical guarantees are model-agnostic—the empirical verification is narrow. 2. **Small evaluation sets and lack of statistical rigor.** The paper reports no confidence intervals, standard deviations, or statistical tests. The LLM-as-judge evaluation (PaLM 2-L for HH and summarization) introduces an additional source of noise: the judge model's preferences may not align perfectly with human preferences, and the paper does not report the judge's agreement rate with the ground-truth human preferences in the Anthropic HH or TL;DR datasets. The preference accuracy in Table 1 (0.709 for Reward-XXS) is on the ground-truth binary preference task, not on the judge model's evaluations. 3. **Missing baselines.** Several comparisons that would strengthen the paper are absent: - CD-Q vs. CD-FUDGE with matched computational budgets during training (number of rollouts, training steps). - A trained-from-scratch multi-objective prefix scorer (trained on weighted reward sum) compared against the post-hoc linear combination in Experiment 4. - Tokenwise CD with different $\lambda$ values fully swept (the tokenwise curves in some figures have only 2–3 points, suggesting sparse $\lambda$ sweeps). - An ablated version of blockwise CD that uses the reward model directly for block ranking (by evaluating the reward model on partial sequences) rather than the prefix scorer, to isolate whether the prefix scorer's approximation error or the blockwise mechanism itself is the bottleneck for matching best-of-K. - Comparison of CD against alternative inference-time control methods beyond FUDGE, such as GeDi (Krause et al., 2021), DIRECTOR (Arora et al., 2022), or NADO (Meng et al., 2022). The paper mentions these in related work but does not empirically compare against them. 4. **KL divergence estimation may be systematically biased in favor of blockwise CD.** The KL upper bounds used for best-of-K and blockwise CD are valid inequalities, but their tightness is unknown for the specific models and prompts used. If the blockwise CD bound is looser (larger overestimate) than the best-of-K bound, blockwise CD could appear to achieve better reward at the same *reported* KL while actually operating at higher *true* KL. The paper does not validate the KL bounds against empirical estimates (e.g., via sampling-based KL approximation). 5. **The gap between CD and best-of-K on noisy rewards remains unexplained.** Table 1 shows that the prefix scorer achieves ~0.63 classification accuracy vs. the reward model's ~0.71, a significant gap that likely drives CD's underperformance on HH and summarization. The paper identifies this as "an area for future investigation" but does not explore concrete remedies: training the prefix scorer with more data, using a better architecture, applying value function learning techniques from the RL literature, or using the reward model itself as the prefix scorer (which would eliminate the training gap at the cost of requiring full decoding to evaluate reward—exactly what the prefix scorer is designed to avoid). A systematic study of how prefix scorer quality scales with training data, model size, and training algorithm would substantially strengthen the paper's practical recommendations. 6. **No latency or throughput measurements.** The paper motivates blockwise CD partly by its streaming and latency advantages over best-of-K (Section 3.2). Yet no wall-clock time, latency distribution, or throughput measurements are reported. The "efficiency" claims are entirely in terms of sample count (K), not actual inference cost. Since blockwise CD requires sequential prefix scorer evaluations between blocks (which may be compute-bound) while best-of-K can parallelize all K decodes, the practical throughput advantage is unclear without system-level benchmarking. **Where the claims hold conditionally:** - The ~8× sample efficiency advantage of blockwise CD (Claim 2) holds for the clean, deterministic length reward (Experiment 1) but not for noisy learned rewards (Experiments 2, 3), where best-of-K maintains an irreducible advantage. - The transfer claim (Claim 3) holds within the PaLM 2 family but is untested across model families. - The multi-objective combination claim (Claim 4) holds for linear combinations of two rewards using CD-FUDGE, but is not demonstrated for CD-Q or for more than two objectives. - The provable optimality claim (Claim 1) holds asymptotically but the empirical gap between theoretical optimality and practical performance (due to value function approximation error) is not characterized. ## 6. Limitations and Trade-offs ### The Prefix Scorer Is a Weaker Discriminator Than the Reward Model It Aims to Replace **The assumption or constraint.** CD's entire mechanism rests on the prefix scorer $V_\theta$ accurately approximating the true value function $V^\star$ of the base policy $\pi_{\text{ref}}$. When used for blockwise ranking, the prefix scorer must reliably discriminate between candidate continuations of varying quality—a task at which it need not match the reward model's raw discriminative power to be useful, but where degradation directly caps performance. The paper explicitly acknowledges this gap in Section 5 (Experiment 2 discussion): the HH prefix scorers achieve only ~0.63 test accuracy on pairwise preference classification versus ~0.71 for the reward model from which they were derived (Table 1), and the authors note this is "likely due to the noisy nature of the training data, and is an area for future investigation to improve the training using value function learning methods better suited to noisy reward environments." The CD-FUDGE variant suffers additionally from high-variance value estimates, as documented in Figure 13 (Appendix B), where CD-FUDGE's length predictions on the Reddit test set are substantially noisier than CD-Q's. **The consequence.** This accuracy gap translates directly into the performance ceiling visible across experiments. For length control (Experiment 1, Figure 3)—a clean, deterministic reward where the prefix scorer makes few errors—blockwise CD-Q essentially matches best-of-K while being more sample-efficient. For helpfulness/harmlessness (Figure 4) and summarization quality (Figure 5)—both noisy, learned rewards—blockwise CD-Q consistently underperforms best-of-K by a margin of 5–8 win-rate percentage points at equivalent KL budgets, and appears to plateau at win rates below 0.72 even at KL values beyond 5, while best-of-K continues to improve. This means that **CD's performance is fundamentally bottlenecked by how well the prefix scorer approximates the reward model**, not by the sampling strategy or the optimization objective. In deployment, a practitioner would need to decide whether the modularity benefits of CD (transfer, multi-objective combination, streaming) justify accepting a permanently lower reward ceiling compared to using the reward model directly for best-of-K ranking. **What evidence exists in the paper.** Table 1 provides the direct comparison of discriminative accuracy (0.624/0.631 for CD-Q vs. 0.804/0.709 for Reward-XXS on train/test splits). Figure 4 shows the downstream consequence: best-of-K reaches win rates of ~0.80 at KL ≈ 8, while blockwise CD-Q saturates around ~0.72. Figure 13 shows the value prediction noise for CD-FUDGE, explaining why blockwise CD-FUDGE collapses (Figures 3 and 4) while tokenwise CD-FUDGE remains functional (since soft reweighting is forgiving of noisy estimates). **Mitigation status.** The paper does not attempt to close this gap. The authors flag it as future work and note that improvements from the deep RL literature (e.g., the Rainbow framework; Hessel et al., 2018) could be applied to CD-Q training, but no concrete remedies are explored. The possibility of using the reward model *directly* as the prefix scorer is not discussed—this would trivially eliminate the accuracy gap at the cost of forward-decoding overhead, since the reward model (trained on complete responses) would need to evaluate partially decoded sequences in a way it wasn't designed for. A systematic study of how prefix scorer quality scales with training data volume, model capacity, and training algorithm is absent. --- ### Only Evaluated on a Single Model Family (PaLM 2), With No Evidence of Cross-Family Transfer **The assumption or constraint.** All experiments use PaLM 2 variants as both the base generative model (XXS in primary experiments, S and XS in transfer experiments) and the architecture for the prefix scorer and reward model (all fine-tuned from PaLM 2-XXS). The paper demonstrates transfer *within* the PaLM 2 family (Experiment 5, Figures 7 and 8), but provides no evidence regarding transfer across model families, architectures, training paradigms, or scales beyond the ~1–2 order-of-magnitude difference between XXS, XS, and S. The paper states in Section 1: "We believe this model is representative of the capabilities of many contemporary LLMs," but provides no supporting evidence for this belief. The theoretical framework (Theorem 2.1) is model-agnostic—any autoregressive LM paired with a learned value function produces the optimal policy—but the practical behavior of CD depends on characteristics of the base model and prefix scorer that may not generalize. **The consequence.** A practitioner considering CD for deployment with models from a different family (LLaMA, GPT, Mistral, etc.) has no empirical basis to predict whether CD will work at all, whether the transfer property will hold, or whether the CD-Q vs. CD-FUDGE performance gap will replicate. Several model-specific factors could plausibly undermine CD's effectiveness: **(1)** the base model's output distribution determines the state distribution on which the prefix scorer is trained—if a different model produces systematically different partial sequences, $V_\theta$ trained on one model may be a poor value estimator for another; **(2)** the quality of the prefix scorer depends on its architecture and initialization, which may interact differently with different base model representations; **(3)** the empirical finding that best-of-K outperforms tokenwise RL (Gao et al., 2023; Rafailov et al., 2023; Yang et al., 2024) is itself observed primarily in specific model families and reward regimes, and may not be universal. Without cross-family validation, CD's claimed generality remains a hypothesis rather than an established fact. **What evidence exists in the paper.** Figures 7 and 8 demonstrate transfer from PaLM 2-XXS to PaLM 2-S and PaLM 2-XS—all members of the same model family, sharing architecture, training data, tokenizer, and pre-training procedure. No experiment tests transfer between distinct model families. The paper does not measure the distributional distance between the base models (e.g., KL divergence between their output distributions) to characterize when transfer succeeds or fails. **Mitigation status.** The paper does not acknowledge this as a limitation. The authors present the transfer result as evidence of CD's robustness without qualifying that it is demonstrated only within a single model family. A systematic study of transfer across model families, with characterization of the conditions under which the value function remains a good proxy, is neither conducted nor proposed as future work. --- ### Practical Overhead of the Prefix Scorer at Inference Time Is Not Quantified **The assumption or constraint.** CD adds computational overhead to every decoding step: for tokenwise sampling, one prefix scorer call per candidate token evaluated at each generation step; for blockwise sampling, $K$ prefix scorer calls per block (one for each candidate continuation of $M$ tokens). The paper motivates CD partly by efficiency considerations—streaming capability, lower latency than best-of-K, reduced sample count for equivalent reward—but reports **zero wall-clock measurements, latency distributions, or throughput benchmarks**. The only "efficiency" metric is sample count $K$, which is an incomplete accounting: generating $K=6$ blocks of $M=32$ tokens with a prefix scorer evaluation between each block may or may not be faster than generating $K=50$ complete sequences in parallel, depending on model sizes, hardware parallelism, memory bandwidth, and the prefix scorer's computational cost relative to the base model. **The consequence.** Without system-level benchmarking, a practitioner cannot evaluate whether CD's operational advantages are real or merely notional. Several hidden costs are plausible: **(1)** The prefix scorer is fine-tuned from the same architecture as the base model (PaLM 2-XXS), meaning it has comparable per-token inference cost. For tokenwise CD, evaluating the prefix scorer on all candidate next tokens at every step could multiply inference cost by a factor proportional to the vocabulary size (or top-$k$ subset). **(2)** Blockwise CD requires sequential prefix scorer evaluations between blocks—while the $K$ candidate blocks within a single step can be decoded in parallel, the prefix scorer evaluation creates a synchronization barrier before the next block can begin. This introduces serial dependencies that best-of-K avoids entirely (all $K$ sequences are independent and fully parallelizable). **(3)** For long responses, blockwise CD makes $\lceil L_x / M \rceil$ rounds of parallel-then-sequential operations, creating latency proportional to response length even with unlimited parallel hardware, whereas best-of-K with sufficient parallelism has latency proportional to the maximum sequence length regardless of $K$. **What evidence exists in the paper.** None. No latency, throughput, or FLOPs measurements are reported anywhere in the paper. The abstract and Section 3.2 make claims about streaming and efficiency that are unbacked by timing measurements. The comparison between blockwise CD-Q ($K=6$) and best-of-K ($K=50$) in Experiment 1 (Figure 3) is framed as an efficiency gain, but this is purely a sample-count comparison—it does not account for the per-block prefix scorer overhead or the sequential blocking structure. **Mitigation status.** The paper does not acknowledge the absence of system-level benchmarking as a limitation. The concluding remarks mention that "a more comprehensive and rigorous understanding of such tradeoffs is left for future work, which might require exploring these methods in conjunction with speculative decoding (Leviathan et al., 2023; Chen et al., 2023; Sun et al., 2023)," but this frames the limitation as a future extension rather than a gap in the current evaluation. --- ### The Tokenwise RL Objective Is More Restrictive Than the Sequence-Level Objective Used by Training-Time Methods, Yet the Relationship Is Not Empirically Characterized **The assumption or constraint.** The paper explicitly acknowledges a structural limitation in Section 2 (Remark): "The tokenwise RL formulation here is more restrictive than the sequence-level RL, used to design RLHF and DPO." This is a deliberate design choice—the tokenwise formulation is what enables the closed-form optimal policy in Theorem 2.1, which is the theoretical foundation for CD. However, the consequence of this restrictiveness is that the optimal policy for the tokenwise objective may be **systematically suboptimal for the sequence-level objective** that the field actually cares about—maximizing expected reward of complete responses subject to a KL penalty on the full sequence distribution. **The consequence.** The paper's empirical results suggest that this restrictiveness is not merely a theoretical concern. Tokenwise CD—which directly implements the optimal tokenwise policy—consistently underperforms blockwise CD and best-of-K (Figures 3, 4, 5) on reward-vs-KL tradeoffs. Best-of-K is an almost-optimal solution to the *sequence-level* KL-regularized RL problem (Yang et al., 2024), and blockwise CD inherits this property by making selections over multi-token horizons. This means that the method the paper theoretically justifies (tokenwise CD) is not the method it empirically recommends (blockwise CD), and the relationship between the two is not fully characterized. A practitioner who implements tokenwise CD—the version with formal guarantees—may be disappointed by its practical performance, especially on noisy learned rewards where tokenwise CD barely improves over the base policy (Figure 4, tokenwise methods at win rate ~0.53). Meanwhile, blockwise CD—which the paper's strongest results rely on—lacks theoretical justification beyond the KL bound and an informal argument that it approximates best-of-K. **What evidence exists in the paper.** Figures 3, 4, and 5 all show blockwise CD dominating tokenwise CD on reward-KL tradeoffs, with the gap being qualitatively larger for noisy rewards (HH, summarization) than for the clean length reward. The paper does not provide a theoretical explanation for this gap, instead citing prior empirical findings (Gao et al., 2023; Rafailov et al., 2023) and the Yang et al. (2024) result on best-of-K optimality. The Remark in Section 2 waves at the issue but does not quantify how much performance is left on the table by the tokenwise formulation. **Mitigation status.** The paper acknowledges the restrictiveness of the tokenwise objective (Section 2 Remark) but does not attempt to characterize the gap between tokenwise-optimal and sequence-optimal policies, either theoretically (under what conditions are they close?) or empirically (by comparing tokenwise CD against a sequence-level RL oracle). The blockwise CD variant is presented as a practical bridge between tokenwise CD and best-of-K, but this is an engineering solution rather than an analytic resolution. A formal analysis of the relationship between the tokenwise and sequence-level objectives—and conditions under which optimizing the former yields good solutions for the latter—remains open. --- ### Multi-Objective Combination Is Demonstrated Only for Blockwise CD-FUDGE With Two Objectives, and the Quality of the Combined Controller Is Not Benchmarked Against Joint Training **The assumption or constraint.** Experiment 4 (Figure 6) demonstrates that independently trained prefix scorers for HH and length can be linearly combined at inference time to trace out a tradeoff curve—a capability the paper presents as a defining advantage over training-time methods. However, this demonstration is narrow in three ways: **(1)** it uses only blockwise CD-FUDGE, not CD-Q (the stronger prefix scorer training method); **(2)** it combines only two objectives (HH and length), using a simple linear combination with configurable weights; **(3)** it does not compare the quality of the combined controller against what a single model trained jointly on the combined reward would achieve, or against what CD-Q would achieve with the same combination. **The consequence.** The implicit claim in the paper's abstract and introduction—that CD solves "a multi-objective RL problem with no additional training" (Abstract)—is partially validated but leaves open critical practical questions. First, the linear combination of independently trained value functions assumes that the value of the combined reward is the weighted sum of the individual value functions. This is true for the *reward* by linearity of expectation (the value function for a weighted sum of rewards is the weighted sum of the individual value functions, under a fixed policy), but it may not hold for *learned* value functions due to approximation errors that interact across objectives. A jointly trained prefix scorer on the combined reward might learn to compensate for these interactions, achieving better calibration. Second, the demonstration is on blockwise CD-FUDGE—the noisier prefix scorer that underperforms CD-Q in single-objective settings (Figures 3, 4). Whether CD-Q's cleaner value estimates would produce better-calibrated multi-objective tradeoffs is unknown. Third, with only two objectives demonstrated, it is unclear whether the approach scales to 3+ objectives without degradation, and how to set the weights in high-dimensional objective spaces without expensive hyperparameter sweeps. **What evidence exists in the paper.** Figure 6 shows the (length, HH win rate) tradeoff for blockwise CD-FUDGE with different linear combination weights. The paper does not report the specific weight values, the number of weight combinations evaluated, or any measure of calibration (e.g., whether the achieved tradeoff corresponds to the intended one). No comparison against a jointly trained multi-objective prefix scorer or a multi-objective PPO/DPO baseline is provided. **Mitigation status.** The paper does not acknowledge these scope limitations for the multi-objective demonstration. The authors present the capability as fully validated ("this experiment would be impossible with training-time KL-regularized RL methods"), which is definitionally true for inference-time combination but does not address whether the achieved tradeoffs are *good* relative to what retraining would produce. Future work on multi-objective value function combination, including calibration studies and scaling to more objectives, is not explicitly proposed. --- ### KL Divergence Estimation Uses Upper Bounds Whose Tightness Is Unknown, Potentially Distorting the Reward-KL Tradeoff Comparisons **The assumption or constraint.** For best-of-K and blockwise CD, the KL divergence between the aligned policy and the base policy cannot be computed exactly because the aligned policy is defined implicitly through a sampling-and-selection procedure, not through an explicit probability distribution. The paper uses upper bounds: $\log(K) - (K-1)/K$ for best-of-K (Stiennon et al., 2020; Beirami et al., 2024) and $\mathbb{E}_{x \sim \mu} (\log(K) - (K-1)/K) \lceil L_x / M \rceil$ for blockwise CD. These bounds are valid inequalities—the true KL cannot exceed the bound—but their *tightness* (how close the bound is to the true KL) depends on the base model's output distribution and the specific prompts, and is not characterized in the paper. For tokenwise CD, PPO, DPO, and IPO, the KL can be estimated more directly from the policy's token probabilities, but the paper does not describe the estimation procedure in sufficient detail to assess its accuracy. **The consequence.** If the KL bounds for best-of-K and blockwise CD are loose to different degrees, the reward-vs-KL tradeoff plots are systematically biased. A method whose KL bound is a looser overestimate will appear to achieve better reward at a given *reported* KL, when it is actually operating at a higher *true* KL (more deviation from the base model) than another method. This could create an illusory advantage for one method over another. The blockwise CD bound involves $\lceil L_x / M \rceil$, which depends on response length and block size—if this term overestimates the per-block KL cost, blockwise CD could appear to dominate tokenwise CD and training-time methods on reward-vs-KL curves while actually deviating more from the base model than reported. The relative tightness of the best-of-K bound and the blockwise CD bound has not been established. **What evidence exists in the paper.** None. The paper uses the bounds without validation against empirical KL estimates (e.g., via Monte Carlo sampling of the log-ratio $\log (\pi(y|x) / \pi_{\text{ref}}(y|x))$). The bounds themselves are cited from prior work, but the prior work's conditions for tightness are not discussed, and the paper does not measure whether those conditions hold for the specific models, prompts, and $K$ values used. The paper states it focuses on KL values smaller than 10 (Section 4.4) because beyond this range policies show signs of overfitting (Eisenstein et al., 2023), but the overfitting threshold itself is presumably based on empirical KL estimates, not bounds, making the comparison potentially inconsistent. **Mitigation status.** The paper does not acknowledge the potential looseness of KL bounds as a limitation. An empirical validation—comparing the analytical bounds against sampling-based KL estimates for a subset of configurations—would substantially strengthen confidence in the tradeoff comparisons but is not conducted. ## 7. Implications and Future Directions ### How This Work Changes the Landscape This paper does not introduce a new alignment algorithm per se—it provides a **theoretical reframing that converts inference-time controlled generation from a collection of heuristics into a principled solution to a well-defined optimization problem**. The magnitude of this shift is best understood as resolving a long-standing ambiguity in the field: are inference-time interventions like FUDGE merely "hacks" that happen to work, or are they doing something mathematically rigorous? The paper's answer is definitive—they are solving the same KL-regularized RL objective as PPO and DPO, just through a different computational pathway (learning a value function rather than updating the generator). This is not a paradigm shift in the sense of upending the dominant alignment paradigm (RLHF remains the central framework), but it is a **substantial reframing** that elevates inference-time methods from second-class approximations to first-class solutions with formal equivalence to training-time methods. **The most important consequence is a clean decoupling of alignment into two separable components: the proposal distribution (the base model) and the value function (the prefix scorer).** Prior to this work, the field largely assumed that alignment required modifying the generator—the proposal distribution had to be shifted toward high-reward regions through PPO, DPO, or IPO. CD demonstrates that equivalent results can be achieved by keeping the generator frozen and learning a separate value function that guides decoding. This decoupling has practical consequences that the paper demonstrates but whose full implications the field has not yet absorbed: you can upgrade the generator without retraining the value function (Experiment 5), combine value functions for different objectives without retraining anything (Experiment 4), and layer CD on top of training-time alignment for compounding gains (Experiment 7). These are not minor conveniences—they are architectural properties that training-time methods fundamentally cannot provide. **The paper reconciles a tension between two empirical observations that had puzzled the community.** On one hand, methods like tokenwise CD and PPO directly optimize the tokenwise or sequence-level RL objective and therefore "should" be optimal. On the other hand, best-of-K—a crude rejection sampling procedure with no explicit optimization—consistently achieves better reward-vs-KL tradeoffs in practice (Gao et al., 2023; Rafailov et al., 2023). This paper does not fully resolve the theoretical puzzle (Yang et al., 2024's analysis of best-of-K optimality is cited but not extended), but it provides the **conceptual bridge**: blockwise CD, which interpolates between tokenwise optimization and sequence-level selection via the block size $M$, inherits best-of-K's empirical advantages while retaining a connection to the RL framework. The operational insight is that evaluating multi-token trajectories (of length $M$) provides more reliable selection signals than per-token value estimates, especially under noisy rewards—a finding with direct practical implications for anyone building controlled generation systems. **The paper also implicitly redirects research attention from generator improvement to verifier improvement.** If the value function is the key component for alignment (as CD's framework implies), then improving the quality and robustness of value function learning becomes the central bottleneck—not improving generator architectures or RL algorithms. The subtext of Table 1 (prefix scorer achieves 0.63 accuracy vs. reward model's 0.71) and the consistent underperformance of CD relative to best-of-K on noisy rewards (Figures 4, 5) is that **value function approximation error is the limiting factor**. This suggests that the field should invest more heavily in techniques from the deep RL literature—better TD learning algorithms, distributional value functions, ensemble methods, uncertainty-aware value estimates—that have been underexplored in the context of language model alignment, where the focus has been on policy optimization rather than value estimation. ### Follow-Up Research This Work Enables **Closing the value function approximation gap between prefix scorers and reward models.** The paper identifies but does not address the central performance bottleneck: CD-Q's prefix scorer achieves 0.63 pairwise preference accuracy versus the reward model's 0.71 (Table 1), and this gap translates directly into CD underperforming best-of-K on HH and summarization (Figures 4, 5). A direct follow-up would train prefix scorers with the same architecture and data but using improvements from the deep RL literature that the paper explicitly mentions but does not explore: distributional RL (learning a distribution over values rather than a point estimate, which could improve ranking calibration), double Q-learning (decoupling action selection from evaluation to reduce overestimation bias), or ensemble methods (training multiple prefix scorers and aggregating their predictions to reduce variance). The experiment would measure whether any of these techniques closes the accuracy gap in Table 1 and whether the downstream blockwise CD performance correspondingly approaches best-of-K. A negative result—showing that the gap persists despite these techniques—would indicate that the value function for natural language rewards is fundamentally harder to learn than the reward model itself, possibly due to the credit assignment problem across long horizons. **Systematic characterization of when value functions transfer across model families.** The paper demonstrates transfer within the PaLM 2 family (Experiment 5, Figures 7, 8) but provides no evidence about cross-family transfer. A rigorous follow-up would train a prefix scorer on one model family (e.g., PaLM 2-XXS) and evaluate it on a different family (e.g., LLaMA-2-7B, Mistral-7B) across multiple reward types (length, sentiment, summarization quality). The key measurement is the correlation between the cross-family performance drop and distributional distance metrics between the base models: KL divergence between their output distributions, embedding space distance, or perplexity of one model's outputs under the other. If the performance drop is small when distributional distance is small (e.g., between similarly-capable models trained on similar data), value function transfer becomes a practical deployment strategy. If the drop is large and unpredictable, CD's modularity advantage is confined to within-family upgrades. A negative result here would be equally informative: it would establish that the value function is tightly coupled to the specific proposal distribution and that CD's transfer claim is limited. **Combining CD with speculative decoding for latency-optimized deployment.** The paper mentions this in its concluding remarks but provides no exploration. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) uses a small draft model to generate candidate tokens that are verified in parallel by the large base model, reducing latency while preserving the base model's output distribution. A natural integration would use the prefix scorer as part of the verification step: instead of simply accepting draft tokens that match the base model's distribution, the verifier could use the prefix scorer to accept or reject draft continuations based on their expected reward. The experiment would measure wall-clock latency, throughput, and reward-vs-KL tradeoffs for this combined system compared to standalone blockwise CD, standalone speculative decoding, and best-of-K. The hypothesis is that speculative decoding's latency benefits compound with CD's alignment benefits, yielding a practical system that is both fast and steerable—exactly the combination needed for production deployment. **Multi-objective CD with 3+ rewards and systematic calibration against joint training.** The paper demonstrates two-objective combination with blockwise CD-FUDGE (Experiment 4, Figure 6) but does not scale to more objectives or benchmark against a jointly trained multi-objective prefix scorer. A rigorous follow-up would train separate prefix scorers for 4–5 distinct objectives (safety, helpfulness, conciseness, factuality, formality), evaluate the quality of post-hoc linear combination across the full Pareto frontier, and compare against a single prefix scorer trained jointly on the weighted sum of all rewards. The key question is whether independently trained value functions can be combined without loss—if the value function for a sum of rewards truly is the sum of individual value functions (as linearity of expectation guarantees for the true $V^\star$), then the learned approximations should combine additively. But approximation errors in each prefix scorer could interact in the sum, causing the combined controller to be miscalibrated (e.g., over-emphasizing one objective relative to the intended weights). The experiment would measure the correlation between intended objective weights and achieved objective tradeoffs, and quantify any degradation relative to joint training. If post-hoc combination works with minimal degradation, CD becomes the default approach for multi-objective alignment. If degradation is substantial, the community needs better methods for value function composition. **Using CD as a diagnostic tool for understanding reward model limitations.** The paper observes that CD's performance degrades on noisy learned rewards (HH, summarization) relative to the clean length reward. This suggests a novel use case: CD can serve as a diagnostic for reward model quality. Because CD separates the value function from the generator, the gap between blockwise CD's performance and best-of-K's performance (which uses the reward model directly) isolates the contribution of value function approximation error. If this gap is large, the reward model is providing signal that the prefix scorer cannot capture—possibly because the reward depends on global properties of the response that are hard to estimate from partial prefixes. A systematic study would train prefix scorers on reward models of varying quality (manipulated by training data volume, model capacity, or label noise), measure the CD-to-best-of-K performance gap for each, and use this to characterize what kinds of rewards are amenable to prefix-scorer-based control versus which require full-sequence evaluation. Rewards that depend on local, prefix-visible properties (toxicity, sentiment, length) should be learnable; rewards that depend on global coherence, logical consistency, or discourse-level features may be inherently resistant to prefix-scorer approximation. This would provide practical guidance on when to use CD versus when to fall back to best-of-K or training-time methods. ### Practical Applications and Downstream Use Cases **Configurable safety filters for deployed chatbots.** A production chatbot serving millions of users needs to balance multiple safety and quality objectives that vary by context: a children's educational app requires strict harmlessness filtering, while a professional writing assistant prioritizes conciseness and factual accuracy. CD enables a single base model paired with multiple independently trained prefix scorers (harmlessness, conciseness, factuality) that are combined at inference time with per-request weights. When a request arrives from the educational app, the system linearly combines the harmlessness scorer with high weight and the others with lower weights; for the writing assistant, it emphasizes conciseness and factuality. The key practical benefit is that adding a new objective or adjusting tradeoffs requires training only a new prefix scorer (a fine-tuning task on modest data) rather than retraining the entire alignment pipeline. Based on the paper's transfer results (Figures 7, 8), the prefix scorers survive base model updates within the same model family, meaning the safety infrastructure can be maintained independently of the underlying language model release cycle. **Streaming alignment for real-time voice assistants.** Voice assistants require low-latency, streaming text generation—the assistant must begin speaking before the full response is computed. Best-of-K is unusable in this setting because it requires generating all K complete responses before any can be served. Blockwise CD, by contrast, makes selection decisions every M tokens (e.g., M=16), enabling streaming with a latency of only M tokens rather than the full response length. At the block size values tested (M=32 yielding the best tradeoffs in Figure 9), the latency overhead is approximately 32 tokens—roughly 1–2 seconds of speech—which is acceptable for conversational applications. Based on Experiment 1 (Figure 3), blockwise CD-Q with K=6 achieves length control comparable to best-of-K with K=50, meaning the assistant can steer response verbosity in real-time with only ~6 parallel decodes per block, a modest computational overhead for a datacenter deployment. **Personalized content generation without per-user model training.** A content platform that generates article summaries, email drafts, or social media posts wants to personalize outputs to individual user preferences—some users prefer detailed summaries, others want bullet points; some want formal tone, others casual. Training-time methods (PPO, DPO) would require maintaining separate model checkpoints per user preference profile, which is infeasible at scale. CD enables a single base model with a library of prefix scorers (length, formality, detail level, sentiment) that are combined with per-user weight vectors at inference time. A new user's preferences can be incorporated by adjusting the weights—no training, no model serving infrastructure changes. The multi-objective demonstration in Experiment 4 (Figure 6) shows that the tradeoff between helpfulness and length can be tuned continuously via weight adjustment, and the linearity of the prefix scorer combination (inherent in the value function's definition under a fixed policy) suggests this extends to arbitrary numbers of objectives. ### When to Prefer This Method The paper articulates a clear tradeoff between CD (specifically blockwise CD-Q) and the two alternative paradigms it competes with: training-time alignment methods (PPO, DPO, IPO) and best-of-K. The decision rule is not universal—it depends on the deployment context—but the paper provides specific conditions that favor each approach: **Prefer blockwise CD-Q when:** - The base model is updated frequently (within the same model family), since the prefix scorer transfers without retraining (Experiment 5, Figures 7, 8), whereas PPO/DPO/IPO would require full retraining for each new checkpoint. - Multiple reward objectives need to be combined or traded off at inference time with per-request configurability, since CD enables linear combination of independently trained prefix scorers with zero additional training (Experiment 4, Figure 6), whereas training-time methods bake a single reward into the generator weights. - Streaming or low-latency generation is required for long-form outputs, since blockwise CD's latency is proportional to block size M (e.g., 32 tokens) rather than full sequence length, unlike best-of-K which must decode all K complete sequences before serving any response (Section 3.2). - A training-time aligned base model already exists and additional inference-time steering is desired, since CD can be layered on top of DPO policies without retraining the prefix scorer (Experiment 7, Figure 10), yielding compounding gains. **Prefer best-of-K when:** - The reward model is substantially more accurate than any prefix scorer can achieve, since best-of-K uses the reward model directly for ranking while CD's performance is capped by prefix scorer quality (Table 1: 0.63 vs. 0.71 accuracy on HH). This is likely to hold for complex, noisy, or globally-dependent rewards where prefix-level value estimation is inherently difficult. - Maximum achievable reward (rather than configurability or latency) is the sole priority, since best-of-K consistently achieves the highest reward-vs-KL tradeoffs across all experiments (Figures 3, 4, 5), and no method in the paper surpasses it on this metric. - Latency is not a constraint and sufficient parallel decoding capacity exists, since best-of-K's operational weakness (full-sequence decoding before selection) is irrelevant in batch processing or offline evaluation settings. **Prefer training-time methods (PPO, DPO, IPO) when:** - Inference-time overhead of any kind is unacceptable, since training-time methods produce aligned models that generate at the same cost as the base model, whereas CD adds prefix scorer evaluations at every block boundary (or every token for tokenwise CD). - The deployment environment cannot support parallel decoding (required for blockwise CD's K candidate blocks), such as on-device inference with strict memory and compute budgets. - The alignment objective is fixed and the base model is stable, since in this scenario training-time methods' lack of configurability is not a disadvantage, and they avoid ongoing inference overhead.