MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions

ArXiv: 2410.02743

🎯 Pitch

Token-level RLHF dilutes rewards across long sequences, slowing learning by up to 2×. MA-RLHF instead groups tokens into macro actions—like phrases or 5-grams—and boosts reward scores by 30% on code and summarization tasks, with zero extra compute.

---

1. Executive Summary

This paper proposes MA-RLHF, a simple yet effective RLHF framework that incorporates macro actions—sequences of tokens or higher-level language constructs (e.g., fixed 5-grams, parsed phrases, or perplexity-bounded chunks)—into the policy optimization process to address the credit assignment problem that plagues standard token-level RLHF over long sequences. By operating at a coarser temporal scale under the semi-Markov Decision Process (SMDP) framework, MA-RLHF reduces the effective decision horizon, yielding more stable policy gradient estimates and achieving 1.7× to 2× faster convergence to a given reward score across text summarization (TL;DR), dialogue generation (HH-RLHF), question answering (WebGPT Comparisons), and code generation (APPS) tasks with Gemma models ranging from 2B to 27B parameters. The approach delivers up to 30% improvement in final reward model scores on summarization and code generation, 18% on dialogue, and 8% on question answering—without introducing additional computational cost at training or inference—while establishing that the benefits of temporal abstraction generalize across tasks and model scales only when the macro-action granularity is appropriately matched to the task structure (e.g., treating the entire sequence as one macro action for summarization versus moderate-length 5- to 10-grams for dialogue).

2. Context and Motivation

The Core Problem: Credit Assignment Over Long Token Sequences

The fundamental problem this paper tackles is the credit assignment problem in token-level RLHF. To understand why this matters, we need to first grasp what standard RLHF does mechanically and why its granularity creates trouble.

In standard RLHF as practiced by InstructGPT (Ouyang et al., 2022) and subsequent work, the language model is treated as a reinforcement learning agent that takes one action per token. When generating a response of, say, 500 tokens, the policy makes 500 sequential decisions. At the end of the sequence, a reward model—trained on human preference comparisons—produces a single scalar score indicating how good the complete response was. The PPO algorithm must then propagate this terminal reward backward through all 500 time steps to determine which individual token decisions contributed positively and which didn't.

This is the temporal credit assignment problem (Kaelbling et al., 1996; Pignatelli et al., 2024): when a reward arrives after many intermediate actions, the learning signal for each action becomes diluted, noisy, and often misleading. A token that was actually critical to a good response (e.g., choosing the right entity in an early sentence) and a token that was irrelevant (e.g., a filler word) both receive similar credit signals because the contributing reward must pass through hundreds of time steps. The variance of the policy gradient estimate scales with the length of the decision horizon, meaning longer sequences produce noisier updates and slower convergence.

This is not merely a theoretical concern—it manifests concretely in several ways:

• Slow convergence: The authors observe that vanilla token-level PPO requires thousands of training steps to achieve a given reward score (visible in Figure 2 and Figure 13, where vanilla PPO curves climb gradually while MA-PPO curves rise sharply).
• Residual low-quality outputs: Figure 10 shows that vanilla PPO leaves a "significant number of low-quality, long-tailed instances" in the RM score distribution, suggesting that some sequences simply don't receive clear enough gradient signals for the model to learn to avoid producing them.
• Training instability at larger scales: The authors note in Section B.2 that training the 7B model with standard PPO on TL;DR required reducing the KL coefficient to 0.01 to avoid instability, a tuning burden that stems partly from noisy gradient estimates.

Why This Problem Is Subtle and Pervasive in Language

The credit assignment challenge in RLHF is amplified by two properties of how modern LLMs process language:

Subword tokenization fragments semantic units. Modern LLMs use byte-pair encoding (BPE; Sennrich et al., 2016) and similar subword tokenization schemes that split words into smaller pieces. The paper points out (Section 1) that OpenAI's tokenizer treats each token as approximately three-quarters of a word on average, making sequences roughly 33% longer than word-level representations would be. This means a single semantically meaningful unit—a word like "unfortunately" or a named entity like "New York"—is often split across multiple tokens, each of which must independently receive credit for the final outcome. The model must learn to coordinate these fragmented sub-word actions when in reality they function as an atomic decision.

Language has local coherence structures that token-level optimization ignores. The paper gives a concrete example (Section 1): the phrase "Big Apple" refers to New York City, but treating "Big" and "Apple" as independent decisions during RL optimization misses the fact that the meaning emerges from the pair together. More generally, language contains multi-token constructions—collocations, idioms, named entities, phrasal verbs, syntactic constituents—whose joint meaning cannot be decomposed into independent token-level contributions. When the RL process assigns credit token-by-token, it has no mechanism to recognize that a group of tokens should be evaluated as a unit. The gradient signal for "Big" and the gradient signal for "Apple" are computed separately, even though their contribution to a good response is intrinsically shared.

This problem is the RL analog of an issue long recognized in language modeling: some sequences are better understood when evaluated holistically. The paper frames MA-RLHF partly as a "de-tokenization" process—reconstructing higher-level language units from subword pieces before computing advantage estimates—thereby aligning the optimization granularity with the natural granularity of linguistic meaning.

Prior Approaches and Their Shortcomings

The paper situates itself relative to several existing lines of work, each of which partially addresses the problem but leaves a gap:

Token-level PPO (the dominant paradigm). Standard RLHF as implemented by Stiennon et al. (2020), Ouyang et al. (2022), and most subsequent systems applies PPO at the token level. This treats the generation process as an MDP with one state per prefix and one action per token. This approach has demonstrated strong results—it is the backbone of InstructGPT, Claude, Gemini, and Llama 2 alignment pipelines—but inherits the full credit assignment burden described above. The paper does not argue that token-level PPO fails, but rather that it is inefficient: it converges slowly and leaves performance on the table that could be captured with better temporal abstraction.

REINFORCE and sequence-level methods (the other extreme). At the opposite end of the spectrum, methods like RLOO (Ahmadian et al., 2024), REINFORCE (Williams, 1992), and GRPO (Shao et al., 2024) treat the entire sequence as a single action, collapsing the MDP into a contextual bandit problem. This completely eliminates the credit assignment problem across tokens—the entire sequence gets a single score—but at the cost of losing all intermediate structure. The policy receives no per-step feedback about which parts of the sequence were on the right track, making the learning signal coarser and potentially slower in a different way. The paper explicitly positions MA-RLHF as a continuum between these extremes (Section 3.2.3): when the macro action length is 1, it's standard token-level PPO; when it's ∞, it's REINFORCE/RLOO. MA-RLHF provides the missing middle ground.

Direct preference optimization (DPO). DPO (Rafailov et al., 2024) and related off-policy alignment methods bypass RL entirely by directly optimizing a preference-based loss from static human comparison data. While DPO avoids the credit assignment problem by not using RL at all, the paper's experiments (Appendix C.3, Table 7) show that MA-PPO substantially outperforms DPO on both TL;DR (RM score of 1.40 vs. 0.03 for DPO) and HH-RLHF (1.55 vs. 0.64), suggesting that online RL with appropriate temporal abstraction captures benefits that offline preference optimization misses—likely because online exploration allows the model to generate and learn from its own distribution-shifted outputs.

Hierarchical RL and options frameworks. The theoretical foundation of macro actions comes from the options framework (Sutton et al., 1999b) and hierarchical RL (Hauskrecht et al., 2013; Mann & Mannor, 2014). These approaches have been studied extensively in robotics, game-playing, and classical planning domains, where they demonstrate faster convergence and better credit assignment over long horizons. However, prior to this work, they had not been applied to RLHF for language model training. The gap is not theoretical—the options framework is well-established—but practical: how do you define meaningful macro actions in the space of natural language tokens without requiring architecture changes or vocabulary retraining?

Fine-grained reward signals. Some prior work has attempted to address credit assignment by providing rewards at a finer granularity. Wu et al. (2023) propose fine-grained human feedback that annotates individual sentences or spans within a response, rather than providing a single overall preference. Tool-augmented reward models (Li et al., 2024) can provide intermediate signals based on external verification. These approaches attack the problem from the reward side (making the reward signal more granular), whereas MA-RLHF attacks it from the policy side (making the policy optimization coarser to match the existing reward granularity). They are complementary—a system could in principle combine macro-action policy optimization with fine-grained rewards—but the paper's contribution is specifically on the policy optimization axis.

How This Paper Positions Itself

MA-RLHF introduces a simple, low-cost mechanism for temporal abstraction that slots directly into existing RLHF pipelines without architectural modifications. The key design constraint—and what distinguishes this from generic options-based approaches—is that the macro actions are defined at the level of the generated sequence during training rather than through vocabulary expansion or model retraining. The policy model continues to output probabilities over individual tokens; the macro-action structure is imposed only during the advantage computation step of PPO. This means:

• No vocabulary changes: Unlike approaches that would require expanding the LLM's vocabulary with multi-token units and retraining from scratch, MA-RLHF maintains the original token-level action space and vocabulary.
• No inference-time overhead: The macro actions affect only how PPO computes advantages during training. At inference, the model generates tokens autoregressively as usual. There is no additional latency cost.
• Drop-in compatibility: MA-PPO can be implemented as a modification to the PPO loss computation (see the code in Appendix E), making it compatible with existing RLHF training stacks like DeepSpeed-Chat (Yao et al., 2023).

The paper positions the various macro-action termination strategies not as competing methods but as different ways to capture different kinds of linguistic structure. Fixed n-gram segmentation captures local co-occurrence patterns; parsing-based segmentation captures grammatical constituency; perplexity-based segmentation captures semantic coherence boundaries where the model's predictive confidence shifts. The empirical finding is that all of them outperform token-level PPO (Figure 5, left), suggesting that any reasonable temporal abstraction is better than none—though the optimal choice depends on the task (Appendix D.2, Table 10: parsing-based termination performs best on HH-RLHF and APPS, while fixed n-grams and n=∞ excel on TL;DR summarization).

The paper also explicitly frames this as a generalization of the RL optimization spectrum (Section 3.2.3): the macro action length parameter $|\omega_\tau|$ interpolates between the fully token-level MDP ($|\omega_\tau| = 1$, equivalent to standard PPO) and the fully sequence-level contextual bandit ($|\omega_\tau| \to \infty$, equivalent to RLOO/REINFORCE/GRPO). The SMDP framework with intermediate $|\omega_\tau|$ values provides a principled way to capture the semi-Markov nature of language—where decisions have variable-duration effects—without committing to either extreme. This is an elegant conceptual unification that explains why macro actions work: they match the temporal abstraction of the optimization to the natural temporal structure of the domain.

3. Technical Approach

3.1 Reader Orientation

MA-RLHF is a training-time modification to the PPO algorithm used in RLHF—not a new model architecture, reward model, or inference procedure. It takes the exact same language model, reward model, and training data as standard RLHF, but changes how advantage estimates are computed during policy optimization: instead of computing advantages at every individual token, it groups tokens into coarser "macro actions" (e.g., fixed 5-grams, syntactically parsed phrases, or perplexity-bounded chunks) and computes a single advantage estimate per group. The system solves the credit assignment problem by reducing the effective number of decision steps between an action and the terminal reward—if a 500-token response previously required propagating credit through 500 individual decisions, grouping tokens into 5-grams reduces this to approximately 100 macro-level decisions, making the learning signal for each decision less noisy and more attributable. The shape of the solution is a semi-Markov decision process (SMDP) wrapper around standard token-level PPO: the underlying policy model still generates tokens autoregressively, the critic still produces per-token value estimates, but the Generalized Advantage Estimation (GAE) and the PPO clipped surrogate loss are applied at the macro-action level rather than the per-token level.

3.2 Big-Picture Architecture (Diagram in Words)

The MA-RLHF system has five components, arranged in a pipeline that modifies a standard PPO training loop at a single insertion point:

1. Policy Model (π_θ): The autoregressive language model being fine-tuned. It generates responses token-by-token given a prompt, producing a sequence of tokens a_0, a_1, ..., a_T and their log-probabilities log π_θ(a_t | a_<t). This is identical to the policy model in standard RLHF—no architecture changes, no vocabulary expansion.

2. Critic Model (V_φ): A value function (typically initialized from the reward model) that produces a scalar value estimate V(s_t) for each prefix state s_t. In MA-RLHF, the critic still operates at the token level—it outputs per-token values—but these per-token values are subsequently aggregated into macro-action values.

3. Reward Model (r_ψ): A frozen model trained on human preference comparisons that scores complete responses. It produces a single scalar reward at the end of the sequence. A per-token KL penalty (measuring divergence from the SFT model) is added to this terminal reward to produce per-token rewards r_t.

4. Macro Action Termination Function (ζ): This is the novel component. Given a generated token sequence, it partitions the tokens into contiguous macro actions ω_1, ω_2, ..., ω_m, where each ω_τ is a subsequence a_{t_τ}, ..., a_{t_{τ+1}-1}. The termination function can use fixed n-grams, randomized n-grams, constituency parsing, or perplexity boundaries (detailed in §3.4.2). This component runs during training only and does not affect inference.

5. MA-PPO Optimizer: The standard PPO clipped surrogate objective and value function loss, but applied at the macro-action level. Given the macro-action boundaries from ζ, the optimizer: (a) aggregates per-token values into per-macro-action values using a weighted combination, (b) aggregates per-token KL-penalized rewards into per-macro-action rewards, (c) runs GAE at the macro-action level to produce advantage estimates Â_τ and target returns Q̂_τ, (d) computes the PPO clipped surrogate loss using macro-action joint log-probabilities log π_θ(ω_τ | s_τ) = Σ_{t=t_τ}^{t_{τ+1}-1} log π_θ(a_t | a_<t), and (e) computes the value loss as ||V(s_τ, ω_τ) - Q̂_τ||².

Information flow: A prompt enters → the policy model generates a response autoregressively → the reward model scores the complete response → per-token KL penalties are added to produce per-token rewards → the critic produces per-token value estimates → the termination function ζ partitions the tokens into macro actions → per-token values and rewards are aggregated into per-macro-action values and rewards → GAE computes macro-level advantages → the PPO clipped objective and value loss are computed at the macro-action level → gradients update both the policy and critic → repeat. The critical path is that the macro-action boundaries are determined after the response is generated but before advantage estimation and loss computation, making this a post-hoc restructuring of the optimization signal rather than a change to the generation process itself.

3.3 Roadmap for the Deep Dive

• First, the formal definition of macro actions—their three components (policy, initiation set, termination condition) and how MA-RLHF instantiates them in the language domain—because this establishes the theoretical framework (SMDPs) and the practical constraint (no vocabulary changes) that drive all subsequent design choices.
• Second, the four termination strategies (fixed n-gram, randomized n-gram, parsing-based, perplexity-based) in detail, including the specific algorithms, hyperparameters, and the linguistic or statistical principles each encodes, since these determine what "temporal abstraction" actually means in practice.
• Third, the MA-PPO optimization procedure—how per-token values and rewards are aggregated into per-macro-action quantities, how GAE is applied at the macro level, and how the PPO clipped surrogate objective and value loss are adapted—because this is where the credit assignment benefit is realized mathematically.
• Fourth, the value function estimation for macro actions, including the three weighting schemes (equal, unit, position-decayed) and their empirical trade-offs, since the aggregation method determines how token-level critic outputs are combined into a coherent macro-level signal.
• Fifth, the connection to the RL optimization spectrum—how varying the macro action length |ω_τ| interpolates between fully token-level MDPs (|ω_τ|=1, standard PPO) and fully sequence-level contextual bandits (|ω_τ|→∞, RLOO/REINFORCE/GRPO)—because this provides the conceptual framework for understanding why macro actions help and how to choose their granularity.
• Sixth, the implementation details and computational cost analysis, including the code-level mechanics of how macro-action boundaries are integrated into the PPO training loop and why the approach introduces no inference-time overhead.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training algorithm paper whose core idea is that credit assignment in token-level RLHF can be improved by restructuring the PPO advantage computation to operate over multi-token macro actions rather than individual tokens, without modifying the underlying language model architecture, vocabulary, or inference procedure.

---

3.4.1 Formal Definition of Macro Actions in the RLHF Context

A macro action (also called an option in the reinforcement learning literature; Sutton et al., 1999b) is formally characterized by three components:

1. A policy π: S × A → [0, 1] that guides action selection within the macro action.
2. A termination condition ζ: S⁺ → [0, 1] that determines when the macro action should end—i.e., after how many primitive actions (tokens) the macro action concludes.
3. An initiation set I ⊆ S specifying which states the macro action can begin from.

In MA-RLHF, two of these three components are simplified to their standard RLHF values:

• The policy π is the same autoregressive language model used in standard token-level RLHF. The policy outputs token-level probabilities π_θ(a_t | a_<t), identical to standard PPO. The macro action does not introduce a separate policy; it only changes how the existing policy's outputs are grouped for optimization.
• The initiation set I is set to all possible token sequences—any state can begin a macro action. This means macro actions can start at any position in the generated text, with no restrictions.
• The termination condition ζ is the only component that MA-RLHF actively designs and varies. It determines the length |ω_τ| of each macro action, i.e., how many consecutive tokens are grouped together.

The consequence of these design choices is that the macro action is entirely defined by its termination condition. In the notation of the paper, a macro action at time step τ is denoted:

$\omega_\tau = \{a_{t_\tau}, a_{t_\tau+1}, \ldots, a_{t_{\tau+1}-1}\}$

where $t_\tau$ is the starting token index of the $\tau$-th macro action and $t_{\tau+1} - 1$ is its final token index. The length of macro action $\tau$ is $|\omega_\tau| = t_{\tau+1} - t_\tau$.

What this notation encodes: A sequence of tokens $a_0, a_1, \ldots, a_T$ is partitioned into $m$ contiguous segments (macro actions) by the termination function. Each macro action $\omega_\tau$ is a subsequence of tokens, and the union of all macro actions covers the entire generated response without gaps or overlaps—the segmentation is a partition.

Why this representation matters: Because the macro actions form a partition of the token sequence, the joint probability of the entire response under the policy can be decomposed either token-by-token or macro-action-by-macro-action:

$\pi_\theta(y | x) = \prod_{t=0}^{T} \pi_\theta(a_t | a_{<t}) = \prod_{\tau=1}^{m} \pi_\theta(\omega_\tau | s_\tau)$

where the macro-action probability is the product of its constituent token probabilities:

$\pi_\theta(\omega_\tau | s_\tau) = \prod_{t=t_\tau}^{t_{\tau+1}-1} \pi_\theta(a_t | a_{<t})$

This equivalence is crucial: it means MA-PPO can compute importance sampling ratios (the π_θ / π_θold terms in the PPO objective) at the macro-action level without changing how the model generates text. The model still produces token-level log-probabilities; those log-probabilities are simply summed within each macro action to obtain the macro-action log-probability. This is what enables MA-RLHF to avoid vocabulary changes—the macro actions are a post-hoc grouping of the existing token-level outputs, not a new action space.

The SMDP framework (Sutton et al., 1999b) provides the theoretical justification. In an MDP, the state transitions occur at every time step. In an SMDP, state transitions occur only at macro-action boundaries, and the reward received between state $s_\tau$ and $s_{\tau+1}$ is the accumulated discounted reward over the duration of the macro action. The SMDP formalism guarantees that standard RL algorithms like PPO remain valid when applied at the macro-action level, provided the rewards are appropriately accumulated and the discounting accounts for the variable duration of each macro action. In MA-RLHF, the discount factor $\gamma$ is set to 1 (as stated in Section 3.2.2), which simplifies the accumulated macro reward to a simple sum of the token-level rewards within the macro action.

The key insight is that this seemingly minor change—grouping tokens into macro actions before computing advantages—has a mathematically grounded variance reduction effect on the policy gradient estimate. The policy gradient for a sequence of length $T$ with a single terminal reward $R$ is:

$\nabla_\theta J = \mathbb{E}\left[\sum_{t=1}^{T} \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot R\right]$

In standard token-level PPO, each of the $T$ terms $\nabla_\theta \log \pi_\theta(a_t|s_t) \cdot R$ contributes variance to the gradient estimate. When using macro actions of average length $n$, the effective number of decision steps reduces to approximately $T/n$, and the gradient becomes:

$\nabla_\theta J = \mathbb{E}\left[\sum_{\tau=1}^{T/n} \nabla_\theta \log \pi_\theta(\omega_\tau | s_\tau) \cdot R\right]$

where $\nabla_\theta \log \pi_\theta(\omega_\tau | s_\tau) = \sum_{t \in \omega_\tau} \nabla_\theta \log \pi_\theta(a_t | s_t)$. The variance of this sum-of-gradients is lower than the variance of the individual token gradients when the token-level gradients within a macro action are correlated (as they typically are for semantically related tokens), because the sum operation averages out some of the independent noise while preserving the shared signal. This is the central mechanism by which MA-RLHF improves learning efficiency: it replaces $T$ noisy gradient contributions with $T/n$ less noisy ones, where each remaining contribution aggregates the gradient signal from multiple tokens that likely contributed jointly to the outcome.

---

3.4.2 The Four Termination Strategies (ζ)

The termination condition $\zeta$ is the sole active design choice in MA-RLHF. It answers the question: given a generated token sequence, where should the boundaries between macro actions fall? The paper explores four strategies, each encoding a different assumption about what constitutes a meaningful linguistic or statistical unit for credit assignment purposes. All four operate post-hoc on the generated sequence—they do not influence generation, only the subsequent advantage computation.

Fixed n-gram Termination

This is the simplest strategy: group tokens into contiguous blocks of exactly $n$ tokens each.

$\omega_\tau = \{a_{t_\tau}, a_{t_\tau+1}, \ldots, a_{t_\tau+n-1}\}$

where $t_{\tau+1} = t_\tau + n$. If the total sequence length $T$ is not divisible by $n$, the final macro action is truncated to contain the remaining tokens.

What it computes: A uniform segmentation of the token sequence into fixed-size chunks, completely agnostic to linguistic content or model behavior. The only parameter is $n$, the chunk size.

Why this works despite its simplicity: Fixed n-grams capture a basic statistical truth about language: adjacent tokens are more likely to be semantically related than distant tokens. Even a content-blind grouping of 5 consecutive tokens will tend to group tokens that belong to the same word, phrase, or local syntactic construction, simply because semantic coherence decays with distance. This means the gradient signals for tokens within a macro action are more likely to share a common cause (e.g., "this chunk is where the model introduced a key entity") than gradient signals for randomly selected token pairs.

Empirical performance: Fixed n-gram termination is described as performing "best" and is used as the default setup (Section 3.2.1). The paper sweeps $n \in \{3, 5, 10, \infty\}$ (Figure 6) and finds that $n=5$ provides the best balance across evaluation dimensions (relevance, coherence, consistency, fluency) as assessed by GPT-4 (Figure 7), while $n=\infty$ achieves the highest RM score on TL;DR summarization specifically (Figure 6, left).

Implementation detail: The $n$ value is a hyperparameter set before training and held constant throughout. The segmentation is deterministic—no randomness in where boundaries fall.

Randomized n-gram Termination

This strategy introduces variability by randomly selecting the length of each macro action from a predefined list.

$\omega_\tau \text{ has length } n_\tau \in \{2, 3, 5, 10\}$

where $n_\tau$ is sampled by shuffling a repeated list of the four allowed lengths (each repeated 3 times, yielding 12 values) and then cycling through the shuffled list to assign lengths to consecutive macro actions. If the total of the assigned lengths does not exactly equal the sequence length, a final macro action of length $\infty$ (i.e., capturing all remaining tokens) absorbs the remainder (Section B.4, point 2).

What it computes: A variable-length segmentation where each macro action's length is drawn from $\{2, 3, 5, 10\}$ without replacement until the list is exhausted, then reshuffled. The distribution is uniform over the four lengths, but the order is randomized.

Why this helps: The variability prevents the model from overfitting to a specific segmentation granularity. If the optimal credit assignment sometimes requires grouping 2 tokens (e.g., a bigram collocation like "New York") and sometimes 10 tokens (e.g., a longer descriptive phrase), a fixed n-gram forces all macro actions to use the same length, which is suboptimal for one of the cases. Randomized n-grams expose the model to multiple granularities during training, allowing the policy gradient to benefit from both fine-grained and coarse-grained credit signals in different parts of the sequence.

Empirical performance: Randomized n-gram termination performs "the best across multiple dimensions, including relevance, coherence, and consistency" according to GPT-4 evaluations (Figure 5, right), though its RM score is comparable to fixed n-gram (Figure 5, left). This suggests that exposure to variable granularity produces more well-rounded improvements in generation quality, even if the raw reward signal is not markedly higher.

Parsing-based Termination

This strategy uses syntactic constituency parsing to define macro actions that align with grammatical structures.

Algorithm (described in Section 3.2.1 and detailed in Appendix B.4):

1. Parse the entire generated response into a constituency tree using a syntactic parser.
2. Perform a depth-first search (DFS) traversal of the tree.
3. At each node, check the number of leaf tokens (terminal symbols) in the subtree rooted at that node.
4. If the number of leaf tokens is less than or equal to a cutoff threshold $C$ (set to $C=5$ in experiments), mark the current position as a macro action boundary—the entire subtree becomes one macro action.
5. If the number of leaf tokens exceeds $C$, recursively descend into the node's children.
6. As a special rule, nodes containing a single token (e.g., punctuation) are merged into the preceding macro action rather than forming their own single-token macro action, avoiding overly fragmented boundaries.

What it computes: Macro actions that correspond to complete syntactic constituents—noun phrases, verb phrases, prepositional phrases, clauses—up to a maximum size of 5 tokens. A macro action ends at the boundary of a syntactic unit, not at an arbitrary fixed count.

Why this is theoretically appealing: In linguistic theory, syntactic constituents are the natural units of composition. A verb phrase like "repair the spaceship" functions as a single semantic unit; splitting it across two macro actions would separate the verb from its object, which likely contributed jointly to the quality of the response. Parsing-based termination aligns the credit assignment granularity with the grammatical granularity of the text, ensuring that tokens that are syntactically bound (e.g., a determiner and its noun, a verb and its direct object) receive shared credit.

Practical difficulty: The paper notes (Appendix B.4, point 3) that discrepancies between the tokenizer used during training and the tokenizer used by the parser can cause mismatches—when the parser's tokenization of the generated text differs from the model's BPE tokenization, the macro action boundaries may not align with the actual token indices. In such cases, MA-RLHF "revert[s] to the standard PPO method," meaning those training examples fall back to token-level optimization, losing the benefit of macro actions for that sample.

Empirical performance: Parsing-based termination shows "promising ability to handle complex grammar" (Section 4.3.1) and performs well on tasks requiring grammatical precision. On the HH-RLHF dialogue task, parsing-based termination achieves an RM score of 1.64 compared to 1.55 for fixed 5-gram (Table 10), and on APPS code generation, it achieves the best overall pass@1 of 5.56 compared to 5.45 for fixed n-gram (Table 9). This suggests that tasks with more structured or formal language benefit from syntactically-informed macro actions, while more free-form tasks like summarization benefit from content-agnostic segmentation.

Perplexity-based Termination

This strategy uses the model's own predictive uncertainty to determine where macro actions should begin and end.

Algorithm (described in Section 3.2.1 and detailed in Appendix B.4, point 4):

1. For each token position $t$ in the generated response, compute the perplexity $p_t$ of the token given the preceding context. This is obtained from the reference model's (SFT model's) logits—no additional forward passes are needed because the logits were already computed during generation.
2. The perplexity of a set of tokens is proportional to the averaged negative log-probability (entropy) of those tokens: $\text{ppl}(\omega_\tau) \propto -\frac{1}{|\omega_\tau|}\sum_{a \in \omega_\tau} \log p_a$.
3. A macro action $\omega_\tau = \{a_{t_\tau}, \ldots, a_{t_{\tau+1}-1}\}$ is constructed by greedily extending the macro action one token at a time, as long as each additional token does not increase the perplexity of the macro action. Mathematically, the macro action terminates when $\text{ppl}(\omega_\tau \cup a_{t_{\tau+1}}) > \text{ppl}(\omega_\tau)$.
4. Additionally, all intermediate prefixes within the macro action must have non-increasing perplexity: $\text{ppl}(\{a_{t_\tau}, \ldots, a_i\}) \geq \text{ppl}(\{a_{t_\tau}, \ldots, a_{i+1}\})$ for all $t_\tau \leq i \leq t_{\tau+1} - 2$. This ensures the perplexity decreases monotonically within each macro action.

What it computes: Macro actions that correspond to spans where the model's prediction confidence is consistently high and improving. The intuition, illustrated in Figure 12, is that within a coherent semantic unit, the model's perplexity should decrease as more context becomes available (e.g., after seeing "the crew from," the model confidently predicts "Mars" with lower perplexity than it predicted the earlier words). When perplexity suddenly increases, it signals a semantic boundary—a shift to a new topic, a new syntactic construction, or a less predictable continuation.

Why this is principled from an information-theoretic perspective: Perplexity is a measure of predictive uncertainty. Tokens within a coherent phrase or clause tend to be highly predictable given their local context, producing low and decreasing perplexity. Boundaries between phrases or clauses tend to introduce new entities, shift topics, or change syntactic structures, producing a perplexity spike. By placing macro action boundaries at perplexity minima (or, equivalently, at the points where perplexity starts increasing), this strategy partitions the sequence into statistically coherent chunks where each chunk's tokens mutually support each other's prediction.

A practical advantage: Because the perplexity is computed from the reference model's logits (which are already available from the generation process), this strategy requires no external parser and no additional forward passes. It is the most computationally lightweight of the linguistically-informed strategies.

Empirical performance: Perplexity-based termination "enhances fluency" (Section 4.3.1) and is described as "most suited for tasks that prioritize smooth and natural language generation" (Figure 5, right). Its overall RM scores are lower than fixed n-gram (1.27 vs. 1.40 on TL;DR, per Figure 5 left), suggesting that while it produces more fluent text, it does not optimize the reward signal as aggressively as content-agnostic chunking. This is consistent with its design: perplexity boundaries capture where the model is confident, not necessarily where credit assignment is most beneficial for reward optimization.

Hyperparameter summary for all termination strategies:

---

3.4.3 MA-PPO: Policy Optimization with Macro Actions

Once the termination function $\zeta$ has partitioned the generated token sequence into macro actions, MA-PPO proceeds through four sequential steps: (1) aggregate per-token values into per-macro-action values, (2) aggregate per-token rewards into per-macro-action rewards, (3) compute macro-level advantages using Generalized Advantage Estimation (GAE), and (4) compute the PPO clipped surrogate objective and value loss at the macro-action level.

Step 1: Value Function Aggregation

The critic model $V_\phi$ produces per-token value estimates $V^\pi(s_t, a_t)$ for every token position $t$ in the generated sequence. MA-RLHF aggregates these into a single macro-action value:

$V^\pi(s_\tau, \omega_\tau) = \sum_{i=0}^{|\omega_\tau|-1} \sigma_{t_\tau + i} \cdot V^\pi(s_{t_\tau + i}, a_{t_\tau + i})$

where $s_\tau$ is the state at the start of macro action $\tau$ (the prefix up to token $t_\tau - 1$), $\omega_\tau$ is the macro action itself (the token subsequence $a_{t_\tau}, \ldots, a_{t_{\tau+1}-1}$), $V^\pi(s_{t_\tau + i}, a_{t_\tau + i})$ is the critic's value estimate for the $i$-th token within the macro action, and $\sigma_{t_\tau + i}$ is a weight controlling the contribution of that token's value to the macro-action value.

What this computes: A weighted sum of the token-level value estimates within the macro action. The weights $\sigma$ determine how much each token's value contributes to the overall macro-action value. The paper explores three weighting schemes (detailed in Appendix D.1, with illustration in Figure 18):

1. Equal assignment ($\sigma_t = 1/|\omega_\tau|$ for all $t$ in $\omega_\tau$): Every token within the macro action contributes equally. This is the default used in all main experiments (Section D.1). It is the simplest scheme and reflects the assumption that all tokens in a macro action share responsibility for the outcome.

2. Unit assignment ($\sigma_t = 0$ for all $t$ except $\sigma_{t_{\tau+1}-1} = 1$): Only the last token of the macro action contributes to its value. This is equivalent to treating the macro action as a single decision made at its final token, with the preceding tokens being "internal" to the macro action. The paper notes this achieves "the best consistency and fluency according to GPT-4 evaluations" (Section D.1, Figure 19, right), suggesting that evaluating a macro action by its endpoint produces more linguistically coherent outputs.

3. Position-decayed assignment ($\sigma_t = 1/((|\omega_\tau| - i) \cdot H)$ where $H = \sum_{j=0}^{|\omega_\tau|-1} 1/(|\omega_\tau| - j)$): Later tokens in the macro action contribute more than earlier tokens, with weights decreasing inversely with distance from the macro action's start. This reflects the intuition that tokens closer to a decision boundary (where the next macro action begins) are more informative about the value of the preceding context.

Why aggregation is necessary: The critic in standard RLHF is trained at the token level; it has not been trained to produce macro-action values. Rather than training a separate macro-level critic (which would require architectural changes), MA-RLHF reuses the existing token-level critic and post-processes its outputs. The aggregation function $\sigma$ serves as a bridge between the token-level critic and the macro-level optimization—it is a hyperparameter of the training procedure, not a learned component.

Step 2: Reward Aggregation

The per-token rewards $r_t$ in RLHF come from two sources: a terminal reward from the reward model (applied only at the final token) and a per-token KL penalty measuring divergence from the SFT model:

$r_t = \underbrace{r_\phi(x, y) \cdot \mathbb{1}[t = T]}_{\text{terminal RM reward}} - \underbrace{\beta \cdot D_{KL}(\pi_\theta(\cdot | s_t) \parallel \pi_{\text{sft}}(\cdot | s_t))}_{\text{per-token KL penalty}}$

where $\beta$ is the KL coefficient (set to 0.05 for 2B models, 0.01 for 7B models on TL;DR; see Table 5 and Section B.2).

The macro-action reward is the sum of the token-level rewards within the macro action, with discount factor $\rho = 1$ (no discounting within a macro action, as stated in Section 3.2.2):

$R_\tau = \sum_{i=0}^{|\omega_\tau|-1} r_{t_\tau + i}$

What this computes: The total reward accumulated during the execution of macro action $\omega_\tau$. Since the terminal RM reward appears only at the final token, most macro actions receive only KL penalty contributions, with the final macro action receiving the RM reward plus its KL penalties.

Why $\rho = 1$: Setting the discount factor to 1 within a macro action means all tokens within the macro action are treated as equally important for the reward computation. This is appropriate because the macro action is being treated as a single unit—if the model decides to produce a 5-gram, all 5 tokens are part of that single decision, and the reward should reflect the combined effect. Discounting within the macro action would artificially down-weight the contribution of later tokens in the macro action, which contradicts the premise that they were part of a single coordinated action.

Step 3: Generalized Advantage Estimation at the Macro Level

With macro-action values $V^\pi(s_\tau, \omega_\tau)$ and macro-action rewards $R_\tau$, MA-PPO applies the standard GAE algorithm (Schulman et al., 2015) at the macro-action time scale. The advantage estimate $\hat{A}_\tau$ and the target return $\hat{Q}_\tau$ are computed using the same GAE formulation as standard PPO, but with the macro-action values and rewards replacing the token-level quantities:

$\hat{A}_\tau = \sum_{l=0}^{m-\tau} (\gamma \lambda)^l \delta_{\tau + l}$

where the TD error $\delta_\tau$ at macro time step $\tau$ is:

$\delta_\tau = R_\tau + \gamma V^\pi(s_{\tau+1}, \omega_{\tau+1}) - V^\pi(s_\tau, \omega_\tau)$

and $\lambda$ is the GAE parameter (set to 0.95 in all experiments, per Table 5), $\gamma$ is the discount factor (set to 1.0), and $m$ is the total number of macro actions in the sequence.

What this computes: The standard GAE advantage, but operating over $m$ macro-action time steps rather than $T$ token-level time steps. The key benefit is that $m \approx T/n$ where $n$ is the average macro action length, so the temporal credit assignment chain is shorter by a factor of $n$. Each $\delta_\tau$ represents the one-step advantage of macro action $\omega_\tau$: how much better the outcome was than predicted by the critic, after accounting for the reward received and the value of the resulting state. The $\lambda$-weighted sum $\hat{A}_\tau$ blends these one-step advantages over the remaining trajectory, with recent steps weighted more heavily (controlled by $\lambda$).

The variance reduction mechanism: The advantage estimate for a token in standard PPO is based on the difference between the actual return and the predicted value, propagated backward through $T$ time steps. This propagation amplifies noise: a small error in the value prediction at step $t$ compounds through all subsequent steps. By reducing the number of propagation steps from $T$ to $T/n$, MA-PPO reduces the compounding of value prediction errors, producing lower-variance advantage estimates. Figure 11 provides empirical evidence: the L2-norm of both advantages and Q-values is consistently lower for MA-PPO than vanilla PPO throughout training, indicating more stable and less noisy estimates.

Step 4: PPO Clipped Surrogate Objective and Value Loss at the Macro Level

The policy loss is the standard PPO clipped surrogate objective, but computed over macro actions:

$\mathcal{L}_{\text{MA-PPO}}(\theta) = \mathbb{E}_\tau \left[ \min\left( \frac{\pi_\theta(\omega_\tau | s_\tau)}{\pi_{\theta_{\text{old}}}(\omega_\tau | s_\tau)} \hat{A}_\tau, \text{clip}\left( \frac{\pi_\theta(\omega_\tau | s_\tau)}{\pi_{\theta_{\text{old}}}(\omega_\tau | s_\tau)}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_\tau \right) \right]$

where:

• $\pi_\theta(\omega_\tau | s_\tau)$ is the probability of macro action $\omega_\tau$ under the current policy, computed as the product of its constituent token probabilities: $\pi_\theta(\omega_\tau | s_\tau) = \prod_{t=t_\tau}^{t_{\tau+1}-1} \pi_\theta(a_t | a_{<t})$
• $\pi_{\theta_{\text{old}}}(\omega_\tau | s_\tau)$ is the same product under the old policy (from the previous iteration)
• $\hat{A}_\tau$ is the macro-level advantage estimate from Step 3
• $\epsilon$ is the clipping parameter (set to 0.2 in all experiments, per Table 5)
• The expectation $\mathbb{E}_\tau$ is taken over macro-action time steps in the training batch

What this computes: The standard clipped PPO objective, where the importance sampling ratio $\pi_\theta / \pi_{\theta_{\text{old}}}$ is computed at the macro-action level rather than the token level. The ratio measures how much the policy has changed for that specific macro action since the last update. The clipping prevents the ratio from deviating beyond $1 \pm \epsilon$, which would indicate an excessively large policy update that could destabilize training. The min operation takes the pessimistic bound: if the advantage is positive and the ratio exceeds $1 + \epsilon$, the clipped ratio is used to prevent the policy from over-optimizing; if the advantage is negative and the ratio falls below $1 - \epsilon$, the clipped ratio is similarly used.

Why the ratio is computed at the macro-action level: The importance sampling correction in PPO ensures that the policy gradient is computed with respect to the distribution that generated the data (the old policy), even though the updates are applied to the current policy. By computing the ratio over the joint probability of a macro action, MA-PPO treats the entire sequence of tokens in $\omega_\tau$ as a single decision that should be reweighted together. If the policy has increased the probability of a beneficial macro action, the ratio will be greater than 1, amplifying the positive advantage signal. If the policy has decreased the probability, the ratio will be less than 1, attenuating or reversing the signal. Crucially, the ratio automatically accounts for the fact that a macro action's probability is the product of its token probabilities: small per-token probability changes compound multiplicatively, producing larger macro-action-level changes, which are then appropriately clipped.

The value loss is the standard mean squared error between the predicted macro-action value and the target return:

$\mathcal{L}_{\text{value}}(\phi) = \mathbb{E}_\tau \left[ \| V^\pi(s_\tau, \omega_\tau) - \hat{Q}_\tau \|^2 \right]$

where $\hat{Q}_\tau$ is the target return from GAE (Step 3). This loss trains the critic to better predict macro-action values, which in turn improves the quality of advantage estimates in subsequent iterations.

Implementation note on the joint probability: In practice, the macro-action log-probability is computed by summing the token-level log-probabilities within the macro action: $\log \pi_\theta(\omega_\tau | s_\tau) = \sum_{t=t_\tau}^{t_{\tau+1}-1} \log \pi_\theta(a_t | a_{<t})$. The importance sampling ratio is then $\exp(\log \pi_\theta(\omega_\tau | s_\tau) - \log \pi_{\theta_{\text{old}}}(\omega_\tau | s_\tau))$. This is implemented in the provided code (Appendix E) by splitting the log-probability difference and the advantage tensor according to the macro-action boundaries, then computing the PPO loss for each macro action segment separately using torch.split.

---

3.4.4 Connection to the RL Optimization Spectrum

MA-RLHF's macro action length parameter $|\omega_\tau|$ defines a continuum between two extremes in RL for language generation (Section 3.2.3):

• When $|\omega_\tau| = 1$ (every token is its own macro action): MA-PPO reduces exactly to standard token-level PPO. The MDP framework applies at the finest granularity. The advantage is computed per token, the importance sampling ratio is per token, and the credit assignment chain has $T$ steps. This is the regime where the variance reduction benefit disappears, but the model retains maximum fine-grained control.

• When $|\omega_\tau| \to \infty$ (the entire sequence is one macro action): MA-PPO converges to REINFORCE (Williams, 1992), RLOO (Ahmadian et al., 2024), or GRPO (Shao et al., 2024). The MDP collapses to a contextual bandit: there is exactly one decision (the entire response), one reward, and no temporal credit assignment within the sequence. The advantage is a single scalar, and the importance sampling ratio is computed over the joint probability of the entire response. This eliminates credit assignment entirely but loses all intermediate structure.

• When $1 < |\omega_\tau| < \infty$: MA-PPO operates in the SMDP regime, providing a middle ground. The credit assignment chain is shorter than token-level PPO by a factor of $|\omega_\tau|$, reducing variance, but the model still receives per-segment feedback rather than losing all intermediate structure.

Empirical evidence for this continuum: Figure 6 shows the effect of varying $n$ in the fixed n-gram termination. On the TL;DR dataset, $n = \infty$ yields the highest RM score, suggesting that for summarization—where the entire response is a holistic summary—treating the sequence as a single macro action provides the best trade-off. On the HH-RLHF dialogue dataset, $n = 10$ performs best, suggesting that dialogue benefits from moderate-length macro actions that can capture turn-level structure without collapsing everything into a single decision. GPT-4 evaluations (Figure 7) show that $n = 5$ provides the best balance across multiple quality dimensions (relevance, coherence, consistency, fluency), while $n = \infty$ shows a fluency advantage specifically. These task-dependent optimal points validate the continuum perspective: there is no universally best $n$; the optimal granularity depends on the temporal structure of the task.

The paper's comparison with RLOO (Appendix C.3, Table 7) provides a concrete data point: on TL;DR with a 2B model, RLOO achieves an RM score of 0.81, standard PPO achieves 0.83, and MA-PPO (n=5) achieves 1.40. This large gap between RLOO (the $n=\infty$ extreme) and MA-PPO (intermediate $n$) suggests that while eliminating credit assignment entirely (RLOO) is slightly worse than full token-level credit assignment (standard PPO), the intermediate SMDP regime substantially outperforms both extremes—at least for this task and model size. The benefit is not simply "coarser is better" but rather "appropriately coarse for the task structure is better."

---

3.4.5 Computational Cost and Implementation

Training cost: MA-RLHF introduces negligible additional computational overhead during training (Section 1: "all without increasing computational complexity during training or inference"). The additional operations are:

1. Macro action boundary computation: For fixed and randomized n-gram termination, this is $O(T)$—a linear scan to partition the sequence. For perplexity-based termination, the perplexities are already computed from logits that were produced during generation, so only a linear scan over the pre-computed perplexity values is needed. For parsing-based termination, the cost of constituency parsing depends on the parser used, but the paper notes that it falls back to standard PPO when parsing fails, limiting the worst-case overhead.

2. Value and reward aggregation: Summing weighted per-token values within macro actions is $O(T)$—linear in sequence length, with a small constant factor.

3. GAE at the macro level: GAE at the macro level operates over $m \approx T/n$ steps rather than $T$ steps, so it is actually faster than token-level GAE by a factor of $n$. The computation is identical to standard GAE but with fewer time steps.

4. Policy loss computation: The policy loss is computed by splitting the log-probability tensors along macro-action boundaries (using torch.split in the provided code, Appendix E). This is also $O(T)$ and involves no additional forward or backward passes through the model.

The dominant cost in RLHF training is the forward and backward passes through the policy and critic models, which process the full sequence regardless of macro-action boundaries. MA-RLHF does not change these passes at all. The operations it adds are all lightweight post-processing of tensors that are already computed. The paper's claim of "no additional computational cost" is therefore accurate in the asymptotic sense—the FLOP count is dominated by model forward/backward passes, and those are unchanged.

Inference cost: At inference time, MA-RLHF has zero overhead. The policy model generates tokens autoregressively with no reference to macro-action boundaries. The termination function, value aggregation, and macro-level GAE are all training-only constructs. The generated checkpoint is a standard autoregressive language model identical to one trained with vanilla PPO.

Implementation in the training loop: The provided PyTorch code (Appendix E) shows the core implementation. The process is:

1. After generating a batch of responses and obtaining per-token log-probabilities, values, rewards, and attention masks, call get_macro_action_positions(start, mask, termination='ngram', n_gram=n_gram) to obtain the list of boundary indices.
2. Call get_macro_action_values(values, mask, start, sequence) and similarly for rewards to aggregate per-token quantities into per-macro-action quantities.
3. Run standard GAE on the aggregated macro-action values and rewards to obtain advantages and returns.
4. Call policy_loss_macro_action(...) which splits the log-probability ratio tensor along macro-action boundaries (using torch.split) and computes the clipped PPO objective per macro action, then averages over all macro actions (weighted by their token counts to handle variable-length segments).
5. Call critic_loss_macro_action(...) which similarly splits the value prediction and computes the MSE loss per macro action.

The key engineering detail is that the macro-action segmentation is recomputed for every training batch from the freshly generated responses. This means the macro-action boundaries adapt as the policy changes during training—they are not fixed based on a pre-computed segmentation of the dataset. This is important because as the policy improves, the linguistic quality of its outputs changes, and the optimal segmentation granularity may shift. The on-the-fly recomputation ensures the macro actions always reflect the current policy's output characteristics.

4. Key Insights and Innovations

Innovation 1: Recasting Credit Assignment as a Temporal Abstraction Problem Rather Than a Reward Design Problem

The dominant prior approaches to the credit assignment problem in RLHF have attacked it from the reward side: design better reward signals. Wu et al. (2023) proposed fine-grained human feedback—annotating individual sentences rather than whole responses. Li et al. (2024) augmented reward models with tool-use signals. Process reward models (Lightman et al., 2023) provide per-step correctness scores for mathematical reasoning. All of these approaches share the same underlying assumption: the bottleneck is that the reward is too coarse, and the fix is to make it finer.

MA-RLHF's most fundamental conceptual move is to invert this framing. Instead of asking "how can we make the reward signal more granular?", it asks "how can we make the policy optimization coarser to match the granularity of the reward we already have?" This is not a minor shift in emphasis—it's a fundamentally different diagnosis of where the bottleneck lies and how to address it.

The diagnosis is grounded in the SMDP framework (Sutton et al., 1999b): the credit assignment problem is not inherently about reward sparsity; it's about the ratio of decision steps to reward events. You can solve it either by increasing the number of reward events (the prior work's approach) or by decreasing the number of effective decision steps (MA-RLHF's approach). Both reduce the temporal distance between an action and the signal that evaluates it, but they do so through opposite mechanisms.

This reframing has practical consequences that make it more than an aesthetic preference. Decreasing decision steps is architecturally cheaper than increasing reward granularity. Fine-grained reward annotation requires expensive human labeling at the span level (Wu et al., 2023) or access to ground-truth intermediate states (Lightman et al., 2023). Process reward models require training auxiliary verifiers on per-step correctness data. MA-RLHF requires none of these—it works with the standard RLHF setup of a single terminal reward per response, using only the existing policy and critic models. The macro-action boundaries are computed algorithmically from the generated text itself, with no additional supervision.

This insight also explains why MA-RLHF's performance gains appear across such different tasks (summarization, dialogue, QA, code generation) with no task-specific tuning beyond the choice of termination strategy. The credit assignment problem is universal—every long-sequence generation task suffers from noisy gradient propagation through many time steps. The temporal abstraction solution is equally universal: reduce the number of time steps. The task-specific question is only how to define macro actions, not whether to use them. The fact that every termination strategy tested (fixed n-gram, randomized, parsing-based, perplexity-based) outperforms token-level PPO (Figure 5, left) supports this: any reasonable temporal abstraction is better than none, because the core benefit—variance reduction through fewer decision steps—is independent of how the boundaries are drawn.

This is a conceptual advance, not merely a new method. It repositions credit assignment from a reward engineering problem to an optimization geometry problem, opening a design space that was largely unexplored in the RLHF literature.

---

Innovation 2: The SMDP Continuum as a Unifying Framework for RL Optimization in Language Generation

Prior to MA-RLHF, the RL-for-language-generation literature was fragmented along an axis that was rarely made explicit: the granularity of the action space. At one extreme, token-level PPO (Ouyang et al., 2022; Stiennon et al., 2020) treats every subword unit as a separate decision, operating in the full MDP regime. At the other extreme, REINFORCE-style methods like RLOO (Ahmadian et al., 2024) and GRPO (Shao et al., 2024) treat the entire sequence as a single action, collapsing to a contextual bandit. DPO (Rafailov et al., 2024) avoids RL entirely by optimizing a preference loss on static data. These approaches were developed and evaluated independently, with no framework for understanding how they relate to each other or how to choose between them for a given task.

MA-RLHF provides this framework through the lens of the macro action length parameter |ω_τ|. Section 3.2.3 makes the connection explicit: when |ω_τ| = 1, MA-PPO is token-level PPO (the MDP); when |ω_τ| → ∞, it converges to RLOO/REINFORCE/GRPO (the contextual bandit); for intermediate values, it operates in the SMDP regime that interpolates between them. This is not just a taxonomic observation—it's a diagnostic tool. It means that the choice between "PPO vs. REINFORCE" or "PPO vs. DPO" is not a binary methodological preference but a point on a continuous spectrum of temporal abstraction, and the optimal point is task-dependent.

The evidence for this task-dependence is in Figure 6: on TL;DR summarization, n = ∞ (fully sequence-level) achieves the highest RM score, while on HH-RLHF dialogue, n = 10 is optimal. On code generation, parsing-based termination performs best (Table 9). These differences are not random noise—they reflect genuine structural differences between the tasks. Summarization produces a single coherent output where the entire response is evaluated holistically, so collapsing to a single macro action makes sense. Dialogue involves multi-turn structure and topic shifts, so moderate-length macro actions that can capture turn-level or utterance-level structure outperform both extremes. Code generation has strong syntactic structure (the program's AST), so syntactically-aligned macro actions provide the best credit assignment.

This continuum perspective also explains a puzzling result in the literature. RLOO (Ahmadian et al., 2024) was proposed as a simpler alternative to PPO that achieves competitive performance with less implementation complexity. But the paper's comparison (Table 7) shows RLOO (RM score 0.81 on TL;DR) underperforming both standard PPO (0.83) and MA-PPO (1.40). The continuum framework explains why: RLOO sits at the |ω_τ| → ∞ extreme, which is suboptimal for tasks that benefit from moderate temporal abstraction but lose important intermediate structure when everything is collapsed to a single decision. The superior performance of intermediate n values (5, 10) shows that the SMDP regime captures benefits that neither extreme can. Prior work comparing PPO vs. REINFORCE was effectively comparing the n=1 and n=∞ endpoints of the spectrum; MA-RLHF shows that the interesting action is in between.

This is a reframing contribution. It doesn't just propose a new method—it provides a conceptual language for comparing existing methods and reasoning about when each is appropriate. Future work can ask "what's the optimal temporal abstraction granularity for this task?" rather than "should I use PPO or REINFORCE?"—a more productive question because the answer is an empirically tunable hyperparameter rather than a binary methodological commitment.

---

Innovation 3: Variance Reduction Through Decision Horizon Compression as the Mechanism—With Direct Empirical Evidence

The paper's claim is that macro actions improve learning by reducing the variance of the policy gradient estimate. This is not a novel theoretical idea—Mann & Mannor (2014) proved that options-based temporal abstraction reduces policy gradient variance in SMDPs, and the intuition that fewer decision steps means less noise is straightforward. What is novel is the paper's combination of (a) connecting this theoretical mechanism to the specific credit assignment pathology in token-level RLHF and (b) providing direct empirical evidence for the variance reduction mechanism rather than just showing improved downstream performance.

Most RLHF papers that propose a new training method demonstrate its benefit through final metrics: higher reward scores, better win rates, faster convergence. These are outcome-level evaluations that don't distinguish why the method works. MA-RLHF goes further by tracking the L2-norm of advantages and Q-values during training (Figure 11). The result is clean: MA-PPO consistently exhibits lower L2-norms for both quantities compared to vanilla PPO throughout training. The advantage L2-norm for MA-PPO stabilizes around 1.5–2.5, while vanilla PPO oscillates between 2.5–5.0. The Q-value L2-norm for MA-PPO stabilizes around 4–6, while vanilla PPO climbs to 8–14.

This is significant because the L2-norm of advantages directly measures the magnitude of the policy gradient updates. In PPO, the policy gradient is proportional to the advantage estimate times the score function. If the advantage estimate has high variance (large L2-norm), the gradient updates are noisy, and training is less stable and less efficient. The lower L2-norms under MA-PPO are direct evidence that the variance reduction mechanism is operating as theorized—fewer decision steps → less compounding of value prediction errors in GAE → lower-variance advantage estimates → more stable gradient updates → faster convergence.

The paper also provides a concrete calculation to ground this intuition (Section 4.5): for a sequence of length T with terminal reward R, the policy gradient has T terms ∇_θ log π(a_t|s_t) · R. Under n-gram macro actions, the effective number of terms reduces to approximately T/n, and each remaining term aggregates the gradients of n tokens. When the token-level gradients within a macro action are positively correlated (as they are for semantically related tokens), the aggregated gradient has the same expected value but lower variance than the sum of independent per-token gradients. The variance reduction factor is approximately n in the ideal case of perfectly correlated token gradients within macro actions, and something less than n in practice.

This evidence-based mechanistic explanation distinguishes MA-RLHF from methods that show improved performance without explaining why. It also provides a diagnostic for practitioners: if you want to know whether macro actions will help for your task, measure the advantage variance during token-level PPO training. If it's high (large L2-norm), temporal abstraction is likely to provide benefits regardless of the specific termination strategy used.

---

Innovation 4: The Finding That the Optimal Temporal Abstraction Is Task-Specific—And That This Specificity Is Detectable Through the SMDP Lens

The paper could have presented a single best macro-action strategy (say, fixed 5-gram) and reported gains across all tasks. This would have been a solid incremental contribution but would have obscured a more interesting finding: the optimal macro-action granularity varies systematically across tasks, and the variation reveals something about the task structure.

The evidence is spread across multiple analyses:

• Summarization (TL;DR): n = ∞ (entire sequence as one macro action) achieves the highest RM score (Figure 6, left). GPT-4 evaluations confirm this: n = ∞ scores highest on fluency, while n = 5 provides the best balance across all dimensions (Figure 7). Summarization is a task where the entire output is evaluated as a single unit—a summary is good or bad as a whole, not sentence-by-sentence in isolation—so collapsing credit assignment entirely makes sense.

• Dialogue (HH-RLHF): n = 10 performs best (Figure 6, right). Dialogue has turn-level structure: a good response addresses the user's query, provides helpful information, and maintains appropriate tone. These are properties that span multiple sentences but don't necessarily require the entire response to be evaluated as one atomic unit. Moderate-length macro actions can capture intra-turn coherence without losing the ability to distinguish between different parts of the response.

• Code generation (APPS): Parsing-based termination achieves the best overall pass@1 (5.56 vs. 5.45 for fixed n-gram; Table 9). Code has strong syntactic structure defined by the programming language's grammar. Aligning macro actions with syntactic constituents (statements, blocks, expressions) provides more meaningful credit assignment than arbitrary n-gram chunking, because the syntactic units correspond to semantically meaningful operations in the program.

• Cross-task comparison of termination strategies (Table 10): On TL;DR, fixed 5-gram achieves an RM score of 1.40 and GPT-4 win rate of 78%; parsing-based achieves 1.37 and 78%. On HH-RLHF, fixed 5-gram achieves 1.55 and 58% win rate, while parsing-based achieves 1.64 and 62%. The relative advantage of parsing-based termination is larger for dialogue than for summarization—dialogue has richer syntactic structure (questions, answers, explanations, clarifications) that benefits more from syntactically-informed macro actions.

This task-specificity is not merely an empirical curiosity. It suggests that the macro-action length can serve as a probe for task structure. A task where n = ∞ is optimal (summarization) is a task where holistic evaluation dominates—the quality of the output is more than the sum of its parts. A task where n = 5 or n = 10 is optimal (dialogue) is a task where multi-token but sub-sequence units matter—phrases, turns, or topical segments. A task where parsing-based termination excels (code generation) is a task with strong formal structure that credit assignment should respect. In principle, one could discover the natural temporal granularity of a new task by sweeping macro-action lengths and observing where performance peaks—a diagnostic procedure enabled by the SMDP framework.

This is an incremental finding in absolute terms (it doesn't fundamentally change how we think about RL), but it's a significant refinement of the macro-action approach. Without this analysis, the takeaway would be "use fixed 5-gram macro actions, they help." With it, the takeaway is "choose your macro-action granularity based on your task's temporal structure, because the optimal granularity varies and the variation is interpretable." The latter is a more actionable and more intellectually satisfying conclusion.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on four datasets spanning diverse generation tasks: TL;DR (Stiennon et al., 2020) for text summarization (93k human-annotated preference pairs for training, 86k for validation, with a portion of validation pairs from CNN Daily Mail as an out-of-distribution test set); Anthropic HH-RLHF (Bai et al., 2022) for dialogue generation (112k training preference pairs, 12.5k validation); WebGPT Comparisons (Nakano et al., 2021) for question answering (19.6k instances, with 5% held out for validation since no separate validation set is provided); and APPS (Hendrycks et al., 2021) for code generation (5k training, 5k test). The data split for SFT / RM / PPO stages allocates 20% of data to SFT, 40% to reward modeling, and 40% to PPO training for the open-ended generation tasks; for APPS (which lacks preference pairs), 80% of data is used for PPO training with both policy and critic initialized from the SFT model (Section 4.1, Appendix B.1–B.2, Table 4).

• Base model(s). The primary experiments use Gemma-2B (Team et al., 2024) as the base model, with scaling experiments extending to Gemma-7B and Gemma-2-27B. For code generation, CodeGemma-1.1-2B and CodeGemma-1.1-7B-it are used. The choice of Gemma models is motivated by their representation of "the capabilities of many contemporary LLMs" and their availability at multiple scales (2B, 7B, 27B), enabling systematic scaling analysis. An additional experiment on Llama-3.2-3B (Appendix C.4) validates generalizability across model families.

• Metrics. For open-ended generation tasks, the paper reports three evaluation types: (1) RM scores on held-out validation sets (2k randomly sampled instances for TL;DR and HH-RLHF; the default validation set for WebGPT); (2) GPT-4 pairwise win rates on 50 randomly drawn instances, evaluating task-specific criteria—relevance, coherence, consistency, and fluency for TL;DR; helpfulness for HH-RLHF; factual accuracy, coherence, and usefulness for WebGPT; (3) human pairwise win rates on the same 50-instance subsets, using analogous criteria. GPT-4 evaluations use gpt-4o-05-13 with randomized response order to mitigate position bias (Appendix F.1). For human evaluation, annotators select preferred responses based on task-specific rubrics: hallucination, verbosity, and overall quality for TL;DR; instruction following and usefulness for HH-RLHF; factual accuracy against retrieved information for WebGPT (Appendix F.2). Inter-rater agreement across 3 annotators on 100 samples averages 68% (64% on TL;DR, 72% on HH-RLHF). For code generation, pass@1 and pass@5 (Chen et al., 2021) are reported on the 5k APPS test set, with breakdowns by difficulty level (Introductory, Interview, Competition). The agreement between RM scores, GPT-4 evaluations, and human evaluations is validated in Table 1: RM–GPT-4 agreement reaches 78%, RM–human agreement reaches 74–76%, and GPT-4–human agreement averages 62% across model sizes.

• Baselines. The paper compares MA-PPO against: (1) Vanilla PPO (token-level PPO as in Ouyang et al., 2022), which is the primary baseline throughout; (2) DPO (Direct Preference Optimization; Rafailov et al., 2024), evaluated on TL;DR and HH-RLHF with Gemma-2B (Appendix C.3, Table 7); (3) RLOO (REINFORCE Leave-One-Out; Ahmadian et al., 2024), evaluated on TL;DR with Gemma-2B (Appendix C.3, Table 7); (4) SFT model (before any RL), serving as the reference point for measuring improvement from RLHF (Table 7, Figure 10). The SFT model is also used as the reference model for KL penalty computation during PPO training.

• Generation budget / compute accounting. The paper measures compute implicitly through training steps—all comparisons between MA-PPO and vanilla PPO are at equal training steps, with convergence speed measured by how many steps are needed to reach a given RM score (e.g., "MA-PPO achieves parity with vanilla PPO approximately 1.7–2 times faster during training," Figure 2). There is no explicit FLOP counting or wall-clock timing. The training hyperparameters (batch size, learning rate, PPO epochs, rollout steps) are held constant between MA-PPO and vanilla PPO for each task and model size (Table 5). The paper emphasizes that MA-RLHF introduces "no additional computational cost during training or inference" (Section 1, abstract), arguing that the added operations (macro-action boundary computation, value aggregation, macro-level GAE) are lightweight post-processing of already-computed tensors. For inference-time evaluation, rejection sampling experiments (best-of-N) use sample sizes N ∈ {4, 8, 16, 32} across temperatures T ∈ {0.2, 0.4, 0.6, 0.8, 1.0, 1.2} (Section 4.4, Figure 8).

• Cross-validation / statistical protocol. RM scores are reported on held-out validation sets (not training data) with shaded regions in Figure 2 representing standard deviation across training runs. GPT-4 and human evaluations use 50-instance subsets; results are reported as win/loss/tie percentages. The paper does not report confidence intervals or statistical significance tests for the pairwise evaluations. For program synthesis, pass@k is computed over the full 5k test set. The scaling experiments (2B → 7B → 27B) use identical training recipes across model sizes with only the KL coefficient adjusted for stability (reduced from 0.05 to 0.01 for the 7B model on TL;DR; Section B.2). The Llama-3.2-3B experiment (Appendix C.4) serves as an out-of-family replication rather than a formal cross-validation procedure.

---

Main Quantitative Results

Summarization (TL;DR): Training Efficiency and Final Quality

Headline result: MA-PPO achieves parity with vanilla PPO approximately 1.7–2× faster in training steps and delivers 30–68% higher final RM scores, with consistent GPT-4 and human preference.

Training convergence (Figure 2): On TL;DR with Gemma-2B, MA-PPO (using fixed 5-gram termination, the default) reaches an RM score equivalent to vanilla PPO's performance at 3.7k steps after only 1.7k steps—a 2.2× speedup. For Gemma-7B, MA-PPO reaches parity with vanilla PPO at approximately 1.9× fewer steps. The RM score curves (Figure 2, left and right panels) show MA-PPO consistently above vanilla PPO throughout training, with the gap widening over time rather than closing. The shaded regions (standard deviation across training runs) indicate that MA-PPO also exhibits lower run-to-run variance.

Final RM scores (Table 2): At the end of training (4.6k steps on TL;DR):

• Gemma-2B: vanilla PPO achieves 0.84, MA-PPO achieves 1.41 (+68%)
• Gemma-7B: vanilla PPO achieves 1.90, MA-PPO achieves 2.47 (+30%)

The 27B scaling experiment (Figure 9, left and mid) confirms the trend: MA-PPO maintains higher RM scores throughout training at this scale as well.

RM score distribution (Figure 3): The distribution of test RM scores for MA-PPO (Gemma-2B, final checkpoint) is shifted rightward relative to vanilla PPO, with fewer low-scoring outputs. MA-PPO concentrates more probability mass at higher RM score values, indicating that the improvement is not just in the mean but in reducing the tail of poor-quality generations.

GPT-4 evaluation (Figure 4, left): On 50 TL;DR test instances, MA-PPO wins against vanilla PPO at rates of 78% (2B model) and 86% (7B model), with tie rates of 16% and 10% respectively. The losses are only 6% (2B) and 4% (7B). This is a strong pairwise preference signal indicating that the RM score improvements translate to quality differences detectable by an LLM judge.

Human evaluation (Figure 4, left): Human annotators prefer MA-PPO outputs over vanilla PPO at rates of 74% (2B) and 69% (7B), with tie rates of 10% and 21%, and loss rates of 16% and 10%. While the human preference margins are slightly smaller than GPT-4's, they remain decisively in MA-PPO's favor. The agreement between GPT-4 and human evaluations (Table 1) is 58% for the 2B model and 64% for the 7B model, suggesting moderate but non-trivial alignment.

Scaling trend (Figure 9, right): Across 2B, 7B, and 27B model sizes, MA-PPO consistently outperforms vanilla PPO on RM scores, GPT-4 win rates, and human win rates. The absolute RM scores increase with model size for both methods, and the gap between MA-PPO and vanilla PPO persists (and may even grow in absolute terms) at larger scales.

---

Dialogue Generation (HH-RLHF): Helpfulness and Harmlessness

Headline result: MA-PPO achieves 18% higher RM scores than vanilla PPO and wins 52–72% of pairwise comparisons, with stronger gains at larger model scales.

Training convergence (Figure 13): On HH-RLHF, MA-PPO reaches parity with vanilla PPO at approximately 2.6k steps (2B) and 3.1k steps (7B), compared to vanilla PPO requiring 5.4k and 5.1k steps respectively—roughly 1.6–2× faster convergence.

Final RM scores (Table 2): At the end of training:

• Gemma-2B: vanilla PPO achieves 1.31, MA-PPO achieves 1.55 (+18%)
• Gemma-7B: vanilla PPO achieves 1.05, MA-PPO achieves 1.24 (+18%)

The consistent +18% improvement across both model sizes suggests the benefit scales proportionally with model capability for dialogue tasks.

RM score distribution (Figure 14): For the 2B model, MA-PPO shifts the RM score distribution rightward, with a higher concentration of scores in the 1.5–2.5 range compared to vanilla PPO. The distribution for MA-PPO has a higher mean and reduced left tail.

GPT-4 evaluation (Figure 4, middle): On 50 HH-RLHF test instances, MA-PPO achieves GPT-4 win rates of 58% (2B) and 72% (7B) against vanilla PPO, with tie rates of 4% and 2%, and loss rates of 38% and 26%. The 7B model shows substantially stronger preference for MA-PPO than the 2B model, suggesting the benefit scales favorably.

Human evaluation (Figure 4, middle): Human annotators prefer MA-PPO at rates of 52% (2B) and 56% (7B), with tie rates of 20% and 24%, and loss rates of 28% and 20%. The narrower human margins compared to GPT-4 (and compared to the TL;DR results) suggest dialogue helpfulness is a more subjective and context-dependent evaluation dimension. The human win rates are above 50% for both scales, confirming a genuine though modest preference.

---

Question Answering (WebGPT Comparisons): Factual Accuracy and Coherence

Headline result: MA-PPO achieves 3–8% higher RM scores than vanilla PPO and wins 64% of GPT-4 pairwise evaluations on the 7B model.

Training convergence (Figure 15): The WebGPT task proved challenging to optimize with both methods—the paper notes that "the policy model exhibited reward hacking behavior which generated repetition tokens to inflate higher reward scores towards the end of training" (Appendix C.2), necessitating early stopping. Despite this, MA-PPO maintains higher RM scores than vanilla PPO throughout training for both 2B and 7B models. The RM score curves for this task are noisier and show more fluctuation than TL;DR or HH-RLHF, consistent with the optimization difficulty.

Final RM scores (Table 2): At early-stopped checkpoints:

• Gemma-2B: vanilla PPO achieves -0.62, MA-PPO achieves -0.60 (+3%)
• Gemma-7B: vanilla PPO achieves -0.61, MA-PPO achieves -0.56 (+8%)

The negative RM scores indicate the reward model rates these outputs below some baseline; the +3–8% improvement represents a reduction in negativity (i.e., outputs are less bad) rather than crossing into positive territory.

RM score distribution (Figure 16): For the 2B model, MA-PPO shifts the distribution rightward from the -0.8 to -0.4 range toward the -0.6 to -0.3 range. The effect is modest but visible.

GPT-4 evaluation (Figure 4, right): On 50 WebGPT test instances, MA-PPO (7B) achieves a GPT-4 win rate of 64% against vanilla PPO across dimensions of factual accuracy, coherence, and usefulness. The 2B model results are not separately reported for GPT-4 on this task. The GPT-4 win rate of 64% is a clear signal, though lower than the 86% and 72% achieved on TL;DR and HH-RLHF respectively.

Human evaluation (Figure 4, right): Human evaluation results for WebGPT are not separately reported in numeric form in the main text, but the right panel of Figure 4 includes WebGPT alongside TL;DR and HH-RLHF in the human evaluation comparison, showing MA-PPO wins consistently across all tasks.

---

Code Generation (APPS): Compiler-Based Rewards

Headline result: MA-PPO achieves 11–35% improvement in pass@1 and 5–30% improvement in pass@5 over vanilla PPO on code generation, with larger gains at the 7B scale.

Pass@1 results (Table 3, upper half): On the full APPS test set:

• CodeGemma-2B: vanilla PPO pass@1 = 4.92, MA-PPO pass@1 = 5.45 (+11%). By difficulty: Introductory 15.26 → 16.56 (+8%), Interview 2.82 → 3.25 (+15%), Competition 0.92 → 0.94 (+2%).
• CodeGemma-7B: vanilla PPO pass@1 = 6.98, MA-PPO pass@1 = 9.48 (+35%). By difficulty: Introductory 20.90 → 26.74 (+28%), Interview 4.26 → 6.22 (+46%), Competition 1.21 → 2.00 (+65%).

The relative gains are larger for the 7B model than the 2B model across all difficulty levels, and the gains increase with problem difficulty—the hardest (Competition) problems see the largest relative improvements (+65% for 7B, though from a very low base of 1.21 → 2.00).

Pass@5 results (Table 3, lower half):

• CodeGemma-2B: vanilla PPO pass@5 = 6.26, MA-PPO pass@5 = 6.60 (+5%). Interview improves from 4.10 to 4.37 (+7%), Introductory from 17.30 to 18.30 (+6%), but Competition decreases from 1.70 to 1.60 (-6%).
• CodeGemma-7B: vanilla PPO pass@5 = 9.06, MA-PPO pass@5 = 11.74 (+30%). Interview improves from 6.57 to 8.37 (+27%), Introductory from 23.30 to 30.30 (+30%), Competition from 2.30 to 3.30 (+43%).

The Competition-level pass@5 regression on the 2B model (-6%) is a notable negative result—it suggests that for the smallest model on the hardest problems, the macro-action approach may not provide benefits at larger sample sizes, possibly because the model's pass@1 is so low (0.92) that sampling more solutions doesn't substantially increase the chance of generating a correct one, and the macro-action variance reduction may not help when no correct solutions exist in the proposal distribution. The gain recovers strongly at the 7B scale (+43% on Competition), consistent with the hypothesis that macro actions amplify existing capability but cannot create it where it's absent.

Reward design for code: Since APPS lacks human preference pairs, no reward model is trained. Instead, the reward is based on compiler signals (Appendix B.5): +1.0 for fully correct code (all unit tests pass), a scaled score between -0.3 and +1.0 proportional to the fraction of unit tests passed for partially correct code, -0.6 for runtime errors, and -1.0 for compile errors. This is a sparse but objective reward that avoids the credit assignment challenges that MA-RLHF is designed to address, yet MA-PPO still provides substantial gains—suggesting the benefits of macro actions extend beyond reward model credit assignment to the more general problem of policy gradient variance.

---

Termination Strategy Comparison

Headline result: All macro-action termination strategies outperform vanilla PPO, but fixed and randomized n-gram strategies achieve the best RM scores and GPT-4 scores, with task-dependent variation in which strategy excels.

RM scores across strategies (Figure 5, left): On TL;DR with Gemma-2B, all five termination strategies (fixed n-gram, randomized n-gram, parsing-based, perplexity-based) consistently outperform vanilla PPO throughout training. By the end of training (4k steps), the ranking by RM score is: fixed n-gram ≈ randomized n-gram > parsing-based > perplexity-based > vanilla PPO. The absolute differences between the top strategies are small—all cluster around 1.3–1.4 RM score vs. vanilla PPO at ~0.8.

GPT-4 evaluation across strategies (Figure 5, right): On the four quality dimensions (relevance, coherence, consistency, fluency), randomized n-gram termination achieves the highest or near-highest scores on relevance, coherence, and consistency. Perplexity-based termination scores highest on fluency. Fixed n-gram and parsing-based termination perform competitively across all dimensions. All macro-action strategies outperform vanilla PPO on all dimensions, with the largest gaps in coherence and relevance.

Cross-task comparison (Table 10): The relative performance of termination strategies varies by task:

• TL;DR: Fixed 5-gram achieves RM score 1.40 with GPT-4 win rate 78%; parsing achieves 1.37 and 78%. The strategies are essentially tied on this task.
• HH-RLHF: Fixed 5-gram achieves RM score 1.55 with GPT-4 win rate 58%; parsing achieves 1.64 and 62%, a clear advantage for parsing-based termination. This suggests dialogue structure benefits from syntactically-informed macro actions.
• APPS (Table 9): Parsing-based termination achieves the best overall pass@1 (5.56) compared to fixed 10-gram (5.45) and PPL-based (5.26). The parsing-based strategy uses a programming-language-aware parser (RedBaron) rather than a natural language parser, aligning macro actions with AST structure.

---

Comparison Against Additional Baselines (DPO and RLOO)

Headline result: MA-PPO substantially outperforms both DPO and RLOO on TL;DR and HH-RLHF, with DPO showing particularly weak results on TL;DR.

RM scores (Table 7, Appendix C.3): On TL;DR with Gemma-2B:

• SFT: -0.64
• DPO: 0.03
• RLOO: 0.81
• Vanilla PPO: 0.83
• MA-PPO (n=5): 1.40

On HH-RLHF with Gemma-2B:

• SFT: 0.13
• DPO: 0.64
• Vanilla PPO: 1.31
• MA-PPO (n=5): 1.55

DPO's poor performance on TL;DR (RM score 0.03 vs. SFT's -0.64, an improvement but far below PPO's 0.83) is striking—it suggests that offline preference optimization from static data may not capture the benefits of online exploration for summarization tasks. DPO performs better on HH-RLHF (0.64 vs. SFT's 0.13) but still substantially below PPO and MA-PPO.

RLOO achieves an RM score of 0.81 on TL;DR, which is slightly below vanilla PPO's 0.83 and far below MA-PPO's 1.40. This means RLOO (which treats the entire sequence as one action—equivalent to MA-PPO with n=∞) is actually marginally worse than standard token-level PPO on this task, while MA-PPO with intermediate n=5 is dramatically better. This is a key piece of evidence for the paper's claim that the SMDP intermediate regime captures benefits neither extreme can.

GPT-4 win rates (Figure 17): On TL;DR, DPO loses to PPO (win rate 50%, loss 44%, tie 6%) and loses heavily to MA-PPO (win 8%, loss 80%, tie 12%). RLOO loses to PPO (win 24%, loss 72%, tie 4%) and loses to MA-PPO (win 8%, loss 88%, tie 4%). On HH-RLHF, DPO loses to PPO (win 34%, loss 52%, tie 14%) and loses to MA-PPO (win 42%, loss 50%, tie 8%). MA-PPO dominates all comparisons, but the margins are smaller on HH-RLHF than TL;DR.

---

Scaling and Generalization Experiments

Headline result: MA-PPO's benefits scale consistently from 2B to 27B parameters, and generalize to the Llama-3.2 model family.

Scaling across model sizes (Figure 9, right): The RM scores, GPT-4 win rates, and human win rates for MA-PPO vs. vanilla PPO are reported across Gemma-2B, 7B, and 27B on TL;DR. MA-PPO outperforms vanilla PPO at all three scales on all three metrics. The absolute RM scores increase with model size (from ~1.4 at 2B to ~3.9 at 27B for MA-PPO; from ~0.8 at 2B to ~3.5 at 27B for vanilla PPO). The relative gap narrows at larger scales (the 27B RM scores differ by ~0.4, compared to ~0.6 at 2B), but this is partly an artifact of the score scale—RM scores are not linearly interpretable across different reward models.

Llama-3.2-3B validation (Appendix C.4, Table 8): On TL;DR, Llama-3.2-3B achieves RM scores of: SFT = 2.38, PPO = 3.33, MA-PPO (n=5) = 3.96. MA-PPO outperforms vanilla PPO by approximately 0.63 RM points. GPT-4 evaluation shows MA-PPO winning 61% of comparisons vs. PPO (with 4% ties, 34% losses). This confirms that MA-RLHF's benefits are not specific to the Gemma architecture or training recipe.

Rejection sampling robustness (Figure 8): MA-PPO demonstrates robust performance across sampling temperatures (T ∈ {0.2, 0.4, 0.6, 0.8, 1.0, 1.2}) and sample sizes (N ∈ {4, 8, 16, 32}) in best-of-N evaluation. While both SFT and vanilla PPO show sensitivity to temperature—with optimal performance at a specific temperature that varies by N—MA-PPO "consistently delivers the best performance at T=1.2 and shows consistent improvement across all tested temperatures" (Section 4.4). The RM score of MA-PPO with best-of-32 at T=1.2 reaches approximately 1.85, compared to vanilla PPO's ~1.35 and SFT's ~1.15 under similar settings. The temperature robustness is particularly evident in Appendix D.4 (Figure 21): vanilla PPO's RM score declines after T > 0.8, while MA-PPO remains stable up to T=1.0.

RM score distribution shift analysis (Figure 10): Comparing the joint distribution of SFT RM scores vs. post-training RM scores:

• Best-of-8 sampling on SFT shifts the distribution up but leaves substantial low-scoring outliers (Figure 10, left).
• Vanilla PPO shifts the distribution further up but still leaves "a significant number of low-quality, long-tailed instances" (Figure 10, mid-left). Many outputs that scored poorly under SFT still score poorly under PPO.
• MA-PPO with n=5 more aggressively shifts the distribution, reducing the prevalence of low-score outliers (Figure 10, mid-right). The correlation between SFT score and MA-PPO score is weaker, indicating MA-PPO is not just amplifying SFT quality but actively improving low-quality outputs.
• MA-PPO with n=∞ (Figure 10, right) shows a similar pattern but with slightly more spread in the mid-range scores.

---

Ablation Studies and Robustness Checks

• Value function aggregation weight schemes (Appendix D.1, Figure 19): Three weighting strategies for aggregating per-token values into macro-action values are compared: equal assignment (uniform weights), unit assignment (only the last token's value), and position-decayed assignment (later tokens weighted more heavily). On TL;DR with Gemma-2B, equal assignment yields the highest RM scores throughout training, while unit assignment achieves the best consistency and fluency according to GPT-4 evaluations. Position-decayed assignment performs intermediately on both metrics. The default equal assignment strikes the best balance for reward optimization, but the unit assignment's fluency advantage suggests that evaluating a macro action primarily by its endpoint may produce more natural language. All three weighting schemes outperform vanilla PPO.

• Varying macro action length n (Figure 6 and Figure 7): On TL;DR with Gemma-2B, sweeping n ∈ {3, 5, 10, ∞} reveals that n = ∞ (entire sequence as one macro action) achieves the highest RM score, with n=5 and n=10 producing similar but slightly lower curves (Figure 6, left). On HH-RLHF, n = 10 achieves the highest RM score (Figure 6, right). GPT-4 evaluations (Figure 7) show that n = 5 provides the best balance across relevance, coherence, and consistency, while n = ∞ scores highest on fluency. This task-dependence is a key finding: summarization (holistic evaluation) benefits from coarser granularity; dialogue (turn-level structure) benefits from moderate granularity.

• Termination strategy cross-task comparison (Table 9 and Table 10): On TL;DR, fixed 5-gram (RM 1.40, GPT-4 win rate 78%) and parsing-based (1.37, 78%) are essentially tied. On HH-RLHF, parsing-based (1.64, 62%) outperforms fixed 5-gram (1.55, 58%). On APPS, parsing-based achieves the best overall pass@1 (5.56 vs. 5.45 for fixed 10-gram, Table 9), though the Interview-level pass@1 for parsing (3.17) slightly underperforms fixed 10-gram (3.25). The task-dependent optimal strategy validates the paper's argument that macro-action granularity should be chosen based on task structure.

• Temperature robustness (Appendix D.4, Figure 21): Evaluating TL;DR outputs at temperatures T ∈ {0.0, 0.2, 0.4, 0.6, 0.8, 1.0}, vanilla PPO's RM score peaks around T=0.4–0.6 and declines after T=0.8, while MA-PPO maintains stable scores across the full temperature range. At T=1.0, MA-PPO (Gemma-2B) achieves an RM score of ~1.4 vs. vanilla PPO's ~1.1. For Gemma-7B, MA-PPO maintains ~2.5 vs. vanilla PPO's decline to ~1.8 at T=1.0. This robustness to sampling temperature is practically important—it means MA-PPO does not require careful temperature tuning at inference time, unlike vanilla PPO.

• Best-of-N rejection sampling (Figure 8): Both MA-PPO and vanilla PPO benefit from best-of-N sampling as N increases from 4 to 32. MA-PPO consistently achieves higher RM scores than vanilla PPO at every (N, T) combination. The SFT model (Figure 8, left) is highly temperature-sensitive, with optimal performance around T=0.8–1.0 depending on N. Vanilla PPO (Figure 8, middle) shows somewhat reduced temperature sensitivity compared to SFT. MA-PPO (Figure 8, right) shows the least temperature sensitivity and the highest absolute scores, with RM scores reaching ~1.85 at N=32, T=1.2.

• RM score distribution comparison: PPO vs. MA-PPO vs. n=∞ (Figure 20): Comparing MA-PPO (n=5) directly against vanilla PPO (Figure 20, mid-right), and MA-PPO (n=∞) against vanilla PPO (Figure 20, right), both show rightward distribution shifts relative to vanilla PPO. The n=5 shift is more concentrated in the high-score region, while n=∞ produces a broader distribution. Best-of-8 on top of MA-PPO (Figure 20, mid-left) shifts the distribution further right compared to raw MA-PPO, confirming that rejection sampling provides additive benefits on top of the RLHF gains.

• L2-norm of advantages and Q-values (Figure 11): The L2-norm of advantages for MA-PPO stabilizes around 1.5–2.5 after an initial spike at the start of training, while vanilla PPO oscillates between 2.5–5.0 with high variance. The L2-norm of Q-values shows a similar pattern: MA-PPO stabilizes around 4–6, vanilla PPO climbs to 8–14. Lower L2-norms indicate more stable and less noisy policy gradient estimates, providing direct evidence for the variance reduction mechanism that the paper claims as the source of MA-RLHF's benefits.

• DPO and RLOO negative results (Table 7, Figure 17): DPO achieves RM score 0.03 on TL;DR (vs. SFT's -0.64 and MA-PPO's 1.40), demonstrating that offline preference optimization is insufficient for this task. RLOO achieves 0.81 on TL;DR (vs. PPO's 0.83 and MA-PPO's 1.40), showing that the sequence-level extreme of the continuum (n=∞) is only marginally better than token-level PPO and far worse than intermediate n=5. Both negative results are informative: they establish that the SMDP intermediate regime is not just "different from extremes" but genuinely superior.

• On-policy vs. off-policy comparison with RLOO (Table 7): RLOO's 0.81 RM score on TL;DR vs. MA-PPO's 1.40 demonstrates that the advantage of macro actions is not simply due to using REINFORCE-style sequence-level optimization. RLOO uses leave-one-out baselines to reduce variance at the sequence level, yet still underperforms MA-PPO's intermediate-granularity approach—suggesting that the temporal abstraction itself (not just the variance reduction technique) provides the benefit.

---

Critical Assessment

The experiments provide strong support for MA-RLHF's central claim that macro-action temporal abstraction improves RLHF training efficiency and final performance, but several important qualifications are necessary when assessing the strength and generalizability of the evidence.

Claim 1: MA-RLHF achieves 1.7× to 2× faster convergence. This claim is well supported by the training curves in Figure 2 (TL;DR), Figure 13 (HH-RLHF), and Figure 15 (WebGPT). The convergence speedup is measured as the training step at which MA-PPO reaches the RM score that vanilla PPO achieves at a later step. On TL;DR with Gemma-2B, this is approximately 1.7k vs. 3.7k steps (2.2×), and on HH-RLHF approximately 2.6k vs. 5.1k steps (2.0×). These are clean, consistent results across tasks. However, the convergence metric is based on RM scores evaluated on a held-out validation set—it is not a measure of downstream task performance convergence. It is possible that RM score convergence speed differs from convergence speed on human judgments or task-specific metrics. The paper does not plot convergence of GPT-4 or human win rates over training steps, so we cannot verify that the 1.7–2× speedup extends to these metrics. Additionally, the convergence comparison is at the granularity of thousands of training steps, which is coarse—a more precise estimate would require interpolation between checkpoints.

Claim 2: MA-RLHF delivers up to 30% improvement in final RM scores on summarization and code generation. This is supported by Table 2 (TL;DR: +30% for 7B, +68% for 2B; code: +35% pass@1 for 7B in Table 3). The +68% figure for TL;DR 2B stands out as substantially larger than other improvements—it's worth noting that this is measured on the absolute RM score, and RM scores are not percentage-scale quantities (the SFT baseline is -0.64, so a move from 0.84 to 1.41 represents a large shift on the RM's internal scale, but interpreting this as "68% better" in a human-meaningful sense is questionable). The +30% on TL;DR 7B and +35% on code pass@1 are more moderate and likely more representative of typical gains. The RM score improvements on HH-RLHF (+18%) and WebGPT (+3–8%) are substantially smaller—the paper's abstract highlights the 30% figure without equally emphasizing the 3–8% range, which is a minor selective reporting concern.

Claim 3: MA-RLHF introduces no additional computational cost during training or inference. This claim is technically accurate in terms of FLOP count—the dominant cost is model forward/backward passes, which are unchanged. However, the paper does not report wall-clock training times, only training steps. The macro-action boundary computation (especially parsing-based) adds latency per training step that is not captured by step-count comparisons. For the default fixed n-gram termination, this latency is negligible, but for parsing-based termination (which requires constituency parsing of every generated response), the per-step overhead could be meaningful. The paper also does not report whether the convergence speedup translates to wall-clock speedup—if MA-PPO takes 1.9× fewer steps but each step is 1.05× slower due to boundary computation, the net wall-clock gain is 1.81×, which is still substantial but not exactly the step-count speedup. For the fixed n-gram default, this distinction is likely academic, but for parsing-based termination, it could be material.

Claim 4: The variance reduction mechanism is responsible for the gains. The L2-norm data in Figure 11 provides correlational evidence: MA-PPO has lower advantage and Q-value L2-norms, and MA-PPO trains faster and better. This is consistent with the variance reduction hypothesis but does not prove causality—it's possible that MA-PPO produces better policies for other reasons, which in turn produce less noisy advantage estimates (reverse causality). A more direct test would be to measure the variance of the policy gradient estimate directly, or to show that the improvement disappears when an alternative variance reduction technique is applied to token-level PPO. The paper's theoretical argument (Section 4.5: reducing the decision horizon from T to T/n reduces gradient variance) is grounded in established RL theory (Mann & Mannor, 2014), but the empirical evidence is primarily the L2-norm correlation. The connection between "lower L2-norm of advantages" and "lower variance of the policy gradient estimate" is intuitive but not formally established in the paper—advantage L2-norm can decrease for reasons unrelated to gradient variance (e.g., if the value function is simply better at predicting returns).

Claim 5: The optimal macro-action granularity is task-dependent. This is one of the paper's most interesting claims and is supported by Figure 6 (n=∞ best for TL;DR, n=10 best for HH-RLHF), Table 9 (parsing-based best for APPS overall), and Table 10 (parsing-based better than fixed n-gram on HH-RLHF but tied on TL;DR). However, the evidence for task-dependence is based on a single sweep of n ∈ {3, 5, 10, ∞} on only two tasks (TL;DR and HH-RLHF). The APPS results use fixed 10-gram and parsing-based termination but don't sweep n values. The WebGPT results don't include n-variation analysis at all. The claim that optimal granularity "varies systematically across tasks" is supported by two data points (summarization wants coarse, dialogue wants moderate). This is suggestive but far from systematic—a more comprehensive analysis would sweep n on all four tasks and would ideally relate the optimal n to measurable task properties (e.g., average response length, syntactic complexity, reward model behavior). The paper's framing of this as a probe for task structure is elegant but would require substantially more evidence to be operationalized as a diagnostic.

Genuine weaknesses in the experimental design:

1. Small evaluation sets for pairwise comparisons. GPT-4 and human evaluations use only 50 instances per task. This is typical for human evaluation studies but means the win rates have substantial sampling error. A 78% win rate on 50 instances (39 wins, 8 ties, 3 losses) has a binomial confidence interval of roughly ±12 percentage points. The reported differences between tasks (e.g., 86% win on TL;DR vs. 72% on HH-RLHF for 7B) may not be statistically distinguishable at this sample size. The paper does not report confidence intervals.

2. RM score as the primary metric for open-ended tasks. The paper relies heavily on RM scores for training convergence comparisons, but RM scores are an imperfect proxy for generation quality—they measure how well the output satisfies the reward model, which is itself an imperfect approximation of human preferences. The agreement between RM and human evaluations (Table 1: 74–76%) is high but not perfect, meaning RM score improvements could partially reflect reward model exploitation rather than genuine quality improvement. The paper acknowledges this risk implicitly by reporting GPT-4 and human evaluations, but the convergence speed claims are based entirely on RM scores.

3. Missing confidence intervals and statistical tests. Throughout the results, the paper reports point estimates (RM scores, win rates, pass@k) without confidence intervals, standard errors, or statistical significance tests. The only exception is Figure 2, where shaded regions represent standard deviation across training runs—but the number of runs is not specified. For the pairwise evaluations, the binomial sampling error from 50 instances is substantial and should be reported.

4. Single reward model per task. The RM score evaluations use a single reward model trained as part of the standard RLHF pipeline. There is no ensembling or cross-validation of reward models. If the reward model has systematic biases, those biases affect all RM score comparisons. The GPT-4 and human evaluations partially mitigate this concern, but only for the final checkpoints—the training convergence analysis relies entirely on the single RM.

5. The SFT baseline is weak for code generation. On APPS, the paper uses compiler-based rewards and initializes both policy and critic from the SFT model (no separate RM training). This means the baseline for measuring improvement is the SFT model itself—but SFT pass@1 for CodeGemma-2B on APPS is not reported, making it impossible to assess how much of the RLHF improvement comes from online exploration vs. the macro-action modification. The comparison is MA-PPO vs. vanilla PPO, both starting from the same SFT initialization, so the relative gain is valid—but an absolute quality assessment relative to the SFT starting point is missing.

6. No combination of macro actions with other variance reduction techniques. The paper compares MA-PPO against vanilla PPO and RLOO individually, but does not test whether macro actions provide benefits on top of other variance reduction methods like GAE with tuned λ, value function clipping, or advantage normalization. If macro actions primarily reduce variance, and these other techniques also reduce variance, the marginal benefit of macro actions might be smaller when combined with a well-tuned baseline PPO implementation. The paper's vanilla PPO uses standard hyperparameters (λ=0.95, γ=1.0, ε=0.2; Table 5), which is a reasonable baseline, but doesn't explore whether more aggressive variance reduction in the baseline would close the gap.

7. Limited scale of human evaluation. Human evaluation uses 50 instances per task, with 3 annotators only for the 7B models on TL;DR and HH-RLHF (all other human evaluations use a single annotator; Appendix F.2). Single-annotator evaluation introduces individual bias—the inter-rater agreement of 68% on 100 samples suggests that different annotators disagree on ~32% of judgments, meaning the reported win rates from single-annotator evaluations could shift meaningfully with different annotators.

Missing experiments that would strengthen the paper:

• Sweep of n-gram length on all four tasks, not just TL;DR and HH-RLHF. This would directly test the claim that optimal granularity is task-dependent and would provide more data points for characterizing the relationship.
• Direct measurement of policy gradient variance, not just advantage L2-norm. Computing the empirical variance of the gradient estimate across minibatches or across random seeds would provide more direct evidence for the claimed mechanism.
• Wall-clock timing of training, to verify that the step-count speedup translates to real-world training speed improvements. This is particularly important for parsing-based termination, which adds per-step latency.
• Evaluation of MA-PPO with alternative base models beyond Gemma and Llama-3.2-3B. The Llama-3.2-3B result (Appendix C.4) is a good start but is limited to TL;DR. Testing on additional model families (e.g., Mistral, Qwen) and on the HH-RLHF and WebGPT tasks would build confidence in generalizability.
• Ablation of the KL penalty coefficient's interaction with macro actions. The paper notes that the 7B TL;DR experiment required reducing the KL coefficient from 0.05 to 0.01 for training stability (Section B.2). It's unclear whether macro actions interact with the KL penalty—since the KL penalty is applied per-token and then summed within macro actions, the effective macro-action KL penalty scales with macro-action length. This interaction could contribute to the observed benefits and should be isolated.
• Analysis of where MA-PPO's gains come from in the output distribution. Figure 10 shows MA-PPO reduces low-quality outliers, but a more detailed analysis—e.g., do the gains come primarily from improving the worst outputs, from making good outputs even better, or from shifting the entire distribution?—would help practitioners understand when MA-RLHF is most valuable. If the gains are concentrated in the left tail, MA-RLHF is primarily a safety/robustness improvement; if they are across the distribution, it's a general quality improvement.

Conditional nature of claims:

The paper's strongest findings—2× convergence speedup, 30%+ final performance gains—hold most clearly on TL;DR summarization with fixed n-gram termination. On other tasks, the gains are more modest: +18% on HH-RLHF, +3–8% on WebGPT, +11–35% on code generation (with the 35% figure applying only to the 7B model). The convergence speedup is demonstrated on TL;DR and HH-RLHF (1.6–2.2×) but not quantified on WebGPT (where training was unstable) or on code generation (where convergence curves aren't shown). The robustness to temperature (Figure 21) is demonstrated only on TL;DR. The scaling to 27B (Figure 9) is demonstrated only on TL;DR. In effect, TL;DR summarization is the most thoroughly validated task, and the strength of evidence degrades as one moves to the other tasks. The paper's general claims ("MA-RLHF achieves 1.7× to 2× faster learning efficiency") are broadly true across the tested conditions but are most rigorously supported for the specific combination of TL;DR + fixed n-gram + Gemma models. Practitioners applying MA-RLHF to new tasks should expect benefits but should calibrate expectations to the +3–18% range seen on dialogue and QA rather than the +30–68% seen on summarization, unless their task structure closely resembles summarization's holistic evaluation pattern.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted for in the Headline Training Efficiency Numbers

The assumption or constraint. The paper's training efficiency claims—1.7× to 2× faster convergence, 30% higher final RM scores—are measured in training steps and implicitly assume zero cost for the macro-action termination computation. The paper states explicitly (Section 1) that MA-RLHF achieves these gains "without introducing additional computational costs during training or inference." Section 3.4.5 argues that the added operations (boundary computation, value aggregation, macro-level GAE) are lightweight O(T) post-processing of already-computed tensors, and therefore the dominant cost (model forward/backward passes) is unchanged.

The consequence. This claim is accurate for fixed n-gram termination—partitioning a sequence into fixed-size chunks requires only a linear scan and adds negligible latency. However, it understates the cost for the more sophisticated termination strategies that the paper presents as alternatives. Parsing-based termination requires running a constituency parser on every generated response in every training batch. The paper provides no measurement of this cost. If the parser adds 10–50% overhead per training step, the convergence speedup of 1.6–2× in steps could translate to a much smaller (or even negative) wall-clock speedup. Even if the parser is highly optimized, the dependency on an external parsing tool introduces a fragility that the paper does not discuss: parser failures (which the paper acknowledges occur, Section B.4: "due to differences between the training and parsing tokenizers, we revert to the standard PPO method when discrepancies occur") mean some fraction of training examples silently fall back to token-level PPO, losing the macro-action benefit for those samples without any diagnostic signal to the practitioner.

Perplexity-based termination is computationally lighter (perplexities are derived from already-computed logits), but it still requires a linear scan with comparisons, and its RM score performance (1.27 on TL;DR, Figure 5 left) is substantially below fixed n-gram (1.40). Randomized n-gram termination adds negligible cost but requires careful implementation of the shuffling and cycling logic described in Appendix B.4.

What evidence exists in the paper. The paper provides no wall-clock time measurements for any training run. All convergence comparisons are in training steps (Figure 2, Figure 13, Figure 15). The parsing-based termination results are presented (Figure 5, Table 10) without any discussion of the computational overhead incurred to produce them. The paper does not report the frequency of parser-tokenizer mismatches that trigger fallback to standard PPO. The claim of "no additional computational cost" in the abstract is therefore true in a narrow technical sense (the asymptotic FLOP count from model forward/backward passes is unchanged) but misleading in a practical sense for the parsing-based and, to a lesser extent, perplexity-based termination strategies.

Mitigation status. The paper does not address this limitation. It makes no recommendation about which termination strategy to use based on computational budget, does not report wall-clock training times, and does not measure the parser's overhead. The default recommendation (fixed n-gram) avoids the issue in practice, but the paper's exploration of more sophisticated strategies implicitly encourages practitioners to try them without warning about the cost. A practitioner reading the paper would reasonably assume that parsing-based termination is a drop-in alternative with similar computational characteristics, which is unlikely to be true.

---

The Approach Provides No Benefit on the Hardest Problems Where Credit Assignment Matters Most

The assumption or constraint. MA-RLHF's core claimed mechanism is improved credit assignment over long sequences—reducing the temporal distance between actions and rewards so that the learning signal for each decision is less noisy. This mechanism predicts that MA-RLHF should provide the largest benefits on tasks with the longest sequences and most delayed rewards, where the credit assignment problem is most severe.

The consequence. The empirical evidence contradicts this prediction in a specific and informative way. On the APPS code generation benchmark (Table 3), the Competition-level problems—which are the longest, most complex, and most challenging instances, and therefore the ones where credit assignment should matter most—show the most fragile results. For the 2B model, Competition pass@5 actually decreases from 1.70 to 1.60 (-6%) under MA-PPO, even as Interview and Introductory problems show gains of 7% and 6% respectively. While the 7B model shows strong Competition-level gains (+43% pass@5), this requires scaling to a model large enough that its base pass@1 is non-trivial (1.21 for Competition). On the TL;DR summarization task, difficulty is not explicitly binned, but the WebGPT question answering task—where both methods struggle and the reward signal is weakest (RM scores remain negative even after training, Table 2)—shows the smallest relative gains from MA-PPO (+3% for 2B, +8% for 7B).

This reveals a fundamental boundary condition: macro actions help credit assignment only when there is genuine credit to assign—that is, when the model is already capable of producing correct or good outputs at some non-trivial rate. If the base model's pass@1 on a problem class is near zero, reducing the variance of the policy gradient does not help because there is no signal to recover from the noise. The variance reduction amplifies existing learning signals but does not create them where they are absent. For the hardest problems, the bottleneck is not credit assignment but capability—the model simply lacks the knowledge or reasoning capacity to produce correct solutions, and no amount of temporal abstraction can compensate for that.

This is directly analogous to the finding in the compute-optimal test-time scaling literature that test-time compute helps on easy-to-medium problems but provides zero benefit on the hardest problems where the base model's pass@1 is near zero. In both cases, the technique (macro actions for training, search for inference) amplifies existing capability but does not create it.

What evidence exists in the paper. Table 3 shows the Competition-level pass@5 regression for the 2B model (-6%) and the substantially smaller relative gains for Competition pass@1 (+2% for 2B) compared to Interview (+15%) and Introductory (+8%). Table 2 shows the small WebGPT gains (+3–8%) on a task where both methods struggle. The paper does not explicitly analyze this difficulty-dependent pattern—it does not bin TL;DR or HH-RLHF results by problem difficulty, does not plot training convergence separately for easy vs. hard instances, and does not discuss why the gains are smallest on the hardest problems. The pattern is visible in the data but not highlighted or explained.

Mitigation status. Not addressed. The paper presents the Competition-level pass@5 regression as a data point without comment. There is no discussion of when MA-RLHF is expected to help vs. not help, no difficulty-binned analysis, and no guidance for practitioners about the capability threshold below which macro actions are unlikely to provide benefits. The parallel to test-time compute scaling limits is not drawn. The limitation is important because it bounds the applicability of the method: MA-RLHF is most valuable for tasks where the model already has some competence (non-trivial pass@1) but struggles with consistent credit assignment, and least valuable for tasks at the frontier of the model's capabilities.

---

The Evaluation Relies Heavily on a Single Reward Model with No Ensembling or Cross-Validation of the Reward Signal

The assumption or constraint. The paper's primary training metric—the RM score used to plot convergence curves (Figure 2, Figure 13, Figure 15) and to report final performance (Table 2)—comes from a single reward model trained on human preference data as part of the standard RLHF pipeline. This reward model is the same one (or a variant initialized similarly) that provides the training signal during PPO. The paper assumes that improvements in this RM score correspond to genuine improvements in generation quality as perceived by humans.

The consequence. There is a well-documented risk in RLHF known as reward over-optimization or reward hacking (Gao et al., 2023): the policy learns to exploit idiosyncrasies of the reward model to achieve high scores without actually improving along the dimensions humans care about. The paper's own evidence on WebGPT (Appendix C.2) demonstrates this risk concretely: "the policy model exhibited reward hacking behavior which generated repetition tokens to inflate higher reward scores towards the end of training," necessitating early stopping. This means the RM scores reported for WebGPT are measured at a checkpoint selected to avoid the worst of the reward hacking, not at a converged optimum—and it is unknown whether similar but subtler reward hacking affects the TL;DR and HH-RLHF results.

More fundamentally, using a single reward model for both training and evaluation creates a circularity problem. The policy is optimized to maximize this specific reward model's scores. If MA-PPO is more effective at optimizing the reward signal than vanilla PPO—which is exactly what the paper claims—then some fraction of the reported RM score improvement may reflect better optimization of the reward model's idiosyncrasies rather than better alignment with human preferences. The paper mitigates this concern by reporting GPT-4 and human evaluations (Figure 4), which show consistent preference for MA-PPO outputs. However, the agreement between RM and human evaluations is 74–76% (Table 1), meaning approximately one-quarter of RM-preferred outputs are not preferred by humans. The convergence speed claims (1.7–2× faster) are based entirely on RM scores—there are no convergence curves for GPT-4 or human win rates over training steps. If RM score convergence is faster than human preference convergence (e.g., because the RM is easier to optimize than true quality), the 1.7–2× speedup figure may overstate the practical training efficiency gain.

What evidence exists in the paper. The WebGPT reward hacking incident is documented in Appendix C.2. Table 1 shows RM-human agreement of 74–76%. The training convergence curves (Figures 2, 13, 15) use only RM scores. The paper acknowledges the RM evaluation limitation implicitly by providing GPT-4 and human evaluations, but these are only for final checkpoints, not for convergence analysis. The paper does not train or evaluate against an ensemble of reward models, does not cross-validate the reward model, and does not report correlation between RM score improvements and human preference improvements at intermediate training checkpoints.

Mitigation status. Partially addressed through GPT-4 and human evaluations at final checkpoints, which confirm that the RM score improvements translate to genuine quality improvements at the end of training. However, the convergence speed claims remain unvalidated by human judgment. A practitioner deploying MA-RLHF would want to know whether the 2× training step reduction translates to a 2× reduction in the time needed to reach human-preferred quality, or whether the RM score converges faster than true quality, yielding a smaller practical speedup. The paper provides no evidence to distinguish these possibilities. Using an ensemble of reward models for evaluation or periodically evaluating with GPT-4 during training would address this limitation.

---

The Approach Has Been Validated on Only Two Model Families (Gemma and Llama-3.2) with No Evidence from the Largest-Scale Models Where RLHF Is Most Commonly Deployed

The assumption or constraint. The paper's experiments use Gemma models at 2B, 7B, and 27B parameters, CodeGemma at 2B and 7B, and a single Llama-3.2-3B validation run on TL;DR only (Appendix C.4). The paper states (Section 4) that Gemma models are "representative of the capabilities of many contemporary LLMs."

The consequence. The largest model tested (Gemma-2-27B) is substantially smaller than the models where RLHF is most commonly deployed in production—LLaMA-70B, LLaMA-405B, GPT-4, Claude, Gemini. At these larger scales, several factors could interact with MA-RLHF's effectiveness in unknown ways. Larger models may have different credit assignment characteristics: their stronger in-context learning and reasoning capabilities might make token-level credit assignment less of a bottleneck naturally, reducing the marginal benefit of macro actions. Alternatively, their longer and more complex outputs might increase the credit assignment problem, amplifying MA-RLHF's benefits. The paper provides no evidence to distinguish these possibilities.

The reliance on Gemma models also means the base model architecture (including the specific tokenizer, which determines average tokens per word and therefore the effective sequence length) is held nearly constant across experiments. The paper's argument that subword tokenization exacerbates credit assignment (Section 1: BPE makes sequences 33% longer than word-level representations) is central to its motivation, but the effect of different tokenizers on MA-RLHF's benefits is never tested. A model with a more aggressive BPE vocabulary (producing shorter sequences) or a character-level tokenizer (producing much longer sequences) might show very different gains. The single Llama-3.2-3B data point (RM score 3.96 for MA-PPO vs. 3.33 for PPO on TL;DR) is encouraging but is limited to one model size, one task, and one termination strategy (n=5).

What evidence exists in the paper. The scaling experiment (Figure 9) covers 2B → 7B → 27B on TL;DR only, showing that the benefit persists but with a potentially narrowing relative gap at larger scales (the 27B RM score difference of ~0.4 is smaller than the 2B difference of ~0.6, though this could be a scale artifact of the RM score metric). The Llama-3.2-3B result (Appendix C.4, Table 8) demonstrates that MA-RLHF generalizes beyond the Gemma architecture for one specific configuration. No experiments use models larger than 27B, no experiments vary the tokenizer, and no experiments test on additional model families (Mistral, Qwen, Phi, etc.) beyond Gemma and Llama-3.2.

Mitigation status. The paper acknowledges the scale limitation in its limitations section (Appendix A): "our experiments are conducted using models with up to 27B parameters; exploring more advanced models, such as LLaMA 3.1 405B or other state-of-the-art architectures and tasks, may provide additional insights into the scalability of MA-RLHF." This is a candid acknowledgment but does not resolve the uncertainty. A practitioner using a 70B+ model cannot confidently predict MA-RLHF's benefits from the current evidence. The paper suggests future work on larger models but provides no preliminary results or theoretical analysis of how the benefits might scale.

---

The Macro-Action Boundaries Are Determined by Fixed Heuristics with No Adaptation to the Policy's Improving Quality During Training

The assumption or constraint. MA-RLHF's termination function ζ is a static, rule-based heuristic: fixed n-gram always uses the same n, parsing-based always uses the same parser and cutoff C=5, perplexity-based always uses the same monotonic decrease criterion. These heuristics are fixed before training and do not adapt as the policy improves. The paper acknowledges this explicitly (Appendix A): "our macro action termination methods are rule-based, including linguistics- or perplexity-driven approaches; future research could explore more complex or learnable termination strategies."

The consequence. As the policy improves during RL training, the characteristics of its generated text change: outputs become more coherent, more reward-aligned, and stylistically different from the SFT starting point. A macro-action segmentation that is optimal for the SFT model's outputs (which may be verbose, disorganized, or repetitive) may be suboptimal for the late-training policy's outputs (which may be concise, well-structured, and fluent). For example, early in training, the policy might produce run-on sentences where fixed 5-gram chunks happen to align well with semantic boundaries by chance; late in training, when sentences become more compact, those same 5-gram boundaries might cut through the middle of phrases, degrading the quality of credit assignment.

The perplexity-based termination strategy partially mitigates this concern because perplexities are computed from the reference model's logits (which are recomputed for each generated sequence), so the macro-action boundaries adapt to the content of each new generation. However, the reference model is the frozen SFT model, not the evolving policy. As the policy diverges from the SFT model (controlled by the KL penalty), the SFT model's perplexity estimates become increasingly off-policy—the SFT model may find the policy's outputs surprising even when those outputs are good, producing perplexity spikes at semantically coherent boundaries simply because the SFT model has never seen that style of generation. The paper does not analyze how the KL divergence between policy and SFT model evolves during training, nor whether the perplexity-based termination remains effective at higher KL divergences.

More fundamentally, the fixed nature of the termination strategies means that MA-RLHF cannot dynamically allocate different granularities to different parts of a sequence. A response might benefit from coarse macro actions in its introductory sentence (where establishing the topic is a holistic decision) and fine-grained macro actions in a detailed explanation (where individual facts need separate credit assignment). The current framework uses the same termination strategy uniformly across the entire sequence, missing this opportunity for adaptive granularity.

What evidence exists in the paper. The paper does not analyze how the optimal macro-action granularity changes over the course of training. The convergence curves (Figure 2, Figure 13) use the same fixed termination strategy throughout. There is no comparison between using a fixed n-gram and using an n-gram whose length is annealed during training (e.g., starting with n=3 early in training and increasing to n=10 later). The randomized n-gram strategy introduces variability but does not adapt—it samples from a fixed distribution throughout training. The paper's analysis of the RM score distribution shift (Figure 10) shows that MA-PPO improves output quality, but does not examine whether the segmentation quality also changes.

Mitigation status. The paper flags learnable or adaptive termination as future work (Appendix A) but does not explore it experimentally. This is a reasonable scope limitation for an initial paper introducing the framework, but it means a practitioner cannot know whether their initial choice of termination strategy and hyperparameters will remain effective throughout training. In practice, the fixed n-gram default (n=5) works well enough across all tested tasks that this limitation may not be practically severe—the convergence curves show consistent improvement throughout training with a single fixed n—but the theoretical concern about off-policy degradation for perplexity-based and parsing-based termination remains unaddressed.

---

The Paper Does Not Resolve the Trade-off Between the Different Macro-Action Termination Strategies, Leaving Practitioners Without Clear Guidance on Which to Choose for a New Task

The assumption or constraint. The paper presents four termination strategies (fixed n-gram, randomized n-gram, parsing-based, perplexity-based) and analyzes their performance across tasks. It finds that all outperform vanilla PPO, but the optimal strategy varies by task: fixed/randomized n-gram works best on TL;DR summarization (Figure 5), parsing-based works best on HH-RLHF dialogue and APPS code generation (Tables 9 and 10), and perplexity-based excels at fluency (Figure 5, right). The paper presents this variation as a finding—"the optimal temporal abstraction is task-specific"—but does not provide a decision procedure for selecting the strategy on a new task.

The consequence. A practitioner approaching a new task (e.g., long-form question answering, creative writing, mathematical reasoning) has four strategies to choose from, each with its own hyperparameters (n-gram length n, parsing cutoff C, etc.), and no principled way to select among them short of running a full sweep. The paper's experiments suggest that the wrong choice can matter: on HH-RLHF, parsing-based achieves a GPT-4 win rate of 62% vs. PPO while fixed 5-gram achieves 58% (Table 10)—a 4 percentage point difference that could be practically meaningful. On TL;DR, the gap between the best (randomized n-gram, RM score ~1.40) and worst (perplexity-based, ~1.27) macro-action strategies is 0.13 RM points, comparable to or larger than the gap between some strategies and vanilla PPO (~0.84 to ~1.27, a 0.43 gap). Choosing the wrong strategy does not eliminate the benefit of macro actions (all strategies beat vanilla PPO), but it leaves performance on the table.

The choice of n in fixed n-gram termination adds another dimension: the paper sweeps n ∈ {3, 5, 10, ∞} on only two tasks (TL;DR and HH-RLHF, Figure 6) and finds different optima (∞ for TL;DR, 10 for HH-RLHF). On APPS and WebGPT, n is not swept. A practitioner on a third task has no basis for choosing n beyond guessing that "moderate values like 5 or 10 provide the best trade-offs" (Section 4.3.2)—a heuristic that is likely reasonable but unvalidated outside the two tested tasks.

More subtly, the paper's framing of this variation as a positive finding—"the optimal macro-action granularity is task-specific and the variation reveals something about the task structure"—assumes the practitioner can afford to discover the optimal granularity through experimentation. For a production RLHF pipeline training a 70B+ model, running multiple training runs to sweep termination strategies and n values may be prohibitively expensive. The paper provides no cheaper proxy for estimating the optimal strategy (e.g., measuring something about the SFT model's outputs that correlates with optimal n, or running a short pilot experiment).

What evidence exists in the paper. The task-dependent optimal strategy is demonstrated in Tables 9 and 10 and Figure 6. The paper does not provide a cross-task analysis that identifies predictive features of the optimal strategy—for example, correlating average response length, syntactic complexity, or reward model characteristics with the best-performing termination method. The paper does not recommend a default strategy for new tasks beyond the observation that fixed n-gram is the default in experiments.

Mitigation status. Partially addressed by the finding that all strategies outperform vanilla PPO, which means the practitioner can pick any reasonable strategy (e.g., fixed 5-gram, the default) and expect some improvement. However, the paper's own results show that the magnitude of improvement varies by strategy and task, and the paper does not provide guidance on how much improvement is being left on the table by choosing the default. The suggestion of learnable termination strategies (Appendix A) would address this in principle—a learned termination function could adapt to the task automatically—but no such method is developed or evaluated. For now, the practical recommendation is implicit but unsatisfying: if you can afford a sweep, do one; if not, use fixed 5-gram and accept that you might not be getting the full benefit.

7. Implications and Future Directions

How This Work Changes the Landscape

MA-RLHF introduces a conceptual reframing rather than a paradigm shift: it repositions the credit assignment problem in RLHF from a reward design bottleneck to an optimization geometry problem solvable through temporal abstraction. This is not a new theoretical discovery—the options framework (Sutton et al., 1999b) and the variance reduction benefits of temporal abstraction (Mann & Mannor, 2014) have been understood in RL for decades—but the paper's contribution is demonstrating that these principles translate cleanly and profitably to the specific setting of language model alignment, where the dominant paradigm has been token-level PPO with no intermediate temporal structure.

The magnitude of the shift is best characterized as a reframing that opens a previously overlooked design dimension. Prior RLHF work optimized along two axes: the reward signal (better reward models, fine-grained feedback, process rewards) and the optimization algorithm (PPO vs. REINFORCE vs. DPO). MA-RLHF introduces a third axis—the temporal granularity of credit assignment—and shows that tuning this axis yields gains comparable to or larger than advances along the other two. The 4× gap between MA-PPO (n=5, RM score 1.40) and DPO (RM score 0.03) on TL;DR (Table 7), and the 1.7× gap between MA-PPO and RLOO (0.81), suggests that temporal abstraction is not a minor tweak but a first-order determinant of RLHF effectiveness for certain tasks.

The work reconciles a latent tension in the RL-for-language-generation literature that was previously unarticulated. Token-level PPO and sequence-level REINFORCE/RLOO have coexisted as alternative approaches with advocates on both sides, but there was no framework for understanding when each is appropriate. MA-RLHF's SMDP continuum (Section 3.2.3) provides that framework: token-level PPO (n=1) is optimal when fine-grained per-token feedback matters; REINFORCE/RLOO (n=∞) is optimal when holistic sequence evaluation dominates; and the SMDP intermediate regime is optimal when the task has multi-token structure that should be evaluated jointly but sub-sequence boundaries that should be distinguished. The empirical finding that the optimal n varies by task (n=∞ for TL;DR summarization, n=10 for HH-RLHF dialogue, parsing-based for APPS code generation) validates this continuum perspective and explains why prior comparisons between PPO and REINFORCE yielded conflicting conclusions—they were testing at different implicit points on the continuum for different tasks.

This reframing makes several research directions more attractive:

• Adaptive temporal abstraction becomes a natural next step. If the optimal macro-action granularity varies by task and possibly by position within a sequence, learning to predict macro-action boundaries from the generated text—rather than using fixed heuristics—could capture task-specific structure automatically. The paper's four termination strategies can be viewed as a first pass at encoding different assumptions about what constitutes a meaningful unit; learned termination could discover units that are optimal for the specific reward model and task distribution.

• Understanding the interaction between temporal abstraction and reward model quality becomes important. The paper shows that all macro-action strategies outperform token-level PPO using the same reward model, but does not explore whether macro actions are more or less sensitive to reward model noise. If macro actions reduce gradient variance, they might be more robust to noisy rewards (because the variance reduction dampens reward noise as well as credit assignment noise). Alternatively, if macro actions amplify the policy's ability to optimize the reward signal, they might also amplify reward hacking—the WebGPT results (Appendix C.2), where both methods exhibited reward hacking, hint at this but don't isolate the macro-action effect.

• The connection between tokenization and credit assignment is empirically validated but not fully explored. The paper's argument that BPE subword tokenization exacerbates credit assignment (Section 1) is central to its motivation, but the experiments use a single tokenizer (Gemma's). Testing MA-RLHF with different tokenizers—character-level, word-level, or morphology-aware—would directly test whether the benefit of macro actions is partly compensating for suboptimal tokenization. If a model with word-level tokenization shows smaller gains from macro actions, it would validate the diagnosis that subword fragmentation is a significant contributor to the credit assignment problem.

Several prior directions become less attractive in light of this work:

• Purely sequence-level REINFORCE methods (RLOO, GRPO) are empirically shown to underperform intermediate-granularity macro actions on the tested tasks (Table 7: RLOO 0.81 vs. MA-PPO 1.40 on TL;DR). This doesn't invalidate these methods—they remain simpler to implement and may suffice for tasks where the holistic evaluation assumption holds—but it suggests that the REINFORCE-vs-PPO debate was asking the wrong question. The right question is "what temporal granularity does this task require?", not "should I use PPO or REINFORCE?"

• Offline preference optimization (DPO) as a replacement for online RLHF is substantially underperformed by MA-PPO on the tested tasks (Table 7: DPO 0.03 vs. MA-PPO 1.40 on TL;DR). While DPO has legitimate advantages (simplicity, stability, no reward model needed), its poor showing on TL;DR—a standard RLHF benchmark—suggests that online exploration with appropriate temporal abstraction captures benefits that offline methods cannot replicate from static preference data alone. The gap is large enough (46× RM score difference) that it warrants skepticism about claims that DPO can fully replace online RLHF for all tasks.

Follow-Up Research This Work Enables

Learned termination functions that predict macro-action boundaries from policy-generated text. The paper's four termination strategies are heuristic proxies for linguistic or statistical coherence—fixed n-grams assume local coherence, parsing assumes syntactic coherence, perplexity assumes predictive coherence. None of these are optimized for the actual objective: improving credit assignment for the specific reward model on the specific task. A natural extension is to train a lightweight boundary predictor—perhaps a small transformer or even a linear classifier on top of the policy model's intermediate representations—that learns to segment sequences into macro actions that maximize the downstream RL optimization efficiency. The training signal could come from the empirical variance of the policy gradient: boundaries that produce lower-variance gradient estimates are better. This would close the gap between the paper's heuristic strategies and what is theoretically achievable, and would automatically adapt the segmentation to the policy's evolving output distribution during training. A strong follow-up would compare learned termination against the best fixed heuristic on the same tasks and model sizes used in the paper, measuring both final RM scores and gradient variance during training. The key question is whether learned boundaries substantially outperform fixed 5-gram (the current default) or whether the heuristic strategies already capture most of the available benefit.

Difficulty-conditioned macro-action granularity within a single training run. The paper's analysis of APPS results (Table 3) reveals that Competition-level problems show smaller or negative gains from MA-PPO compared to Introductory and Interview problems. This mirrors the finding in test-time compute scaling that hard problems benefit less from additional inference computation because the base model lacks the capability to produce correct solutions. A natural hypothesis is that macro-action granularity should be difficulty-dependent: easy problems (where the model has high pass@1) might benefit from coarse macro actions (larger n) because credit assignment is the main bottleneck, while hard problems (where the model's pass@1 is near zero) might benefit from finer granularity or even token-level PPO because the optimization needs to extract every possible learning signal from the rare correct outputs. A strong follow-up would bin training data by difficulty (using the SFT model's pass@1 as a proxy, similar to the oracle difficulty estimation in the test-time compute scaling literature), train MA-PPO with difficulty-dependent n values, and compare against uniform n. The specific prediction is that the performance gap between MA-PPO and vanilla PPO is largest on easy problems and smallest or zero on the hardest problems, and that using n=1 for hard problems while using n=5 or n=∞ for easy problems outperforms a uniform n strategy. This would directly connect the temporal abstraction literature to the difficulty-conditioned scaling literature.

Interaction between macro actions and KL penalty strength. The paper notes that training the 7B model on TL;DR required reducing the KL coefficient from 0.05 to 0.01 for stability (Section B.2), but does not analyze whether this interacts with macro actions. Since the KL penalty is applied per-token and summed within macro actions, the effective macro-action-level KL penalty scales with macro-action length: a macro action of length 5 receives 5× the KL penalty of a single token. This means macro actions implicitly increase the KL penalty pressure on the policy—each macro-action decision is penalized more heavily for deviating from the SFT model than each token-level decision. This could be beneficial (stronger regularization prevents reward hacking) or harmful (excessive penalty prevents the policy from exploring useful divergences from the SFT). A strong follow-up would systematically vary the KL coefficient across different macro-action lengths and measure the interaction. The hypothesis is that larger n values require smaller β to maintain the same effective regularization strength, and that tuning β to account for macro-action length could recover additional performance. Conversely, if MA-PPO's robustness to temperature (Figure 21) is partly due to this implicit KL amplification, reducing β might reduce that robustness—an important trade-off for practitioners to understand.

Macro actions combined with process reward models or step-level feedback. The paper studies MA-RLHF with standard terminal-only reward models. However, recent work on process reward models (PRMs; Lightman et al., 2023) and fine-grained human feedback (Wu et al., 2023) provides per-step or per-span reward signals that partially address the credit assignment problem from the reward side. Macro actions attack the same problem from the optimization side. These approaches should be complementary: per-step rewards reduce the temporal distance from the reward perspective, while macro actions reduce the temporal distance from the policy gradient perspective. A strong follow-up would combine MA-PPO with a process reward model on a task like mathematical reasoning (where PRMs are well-established) and measure whether the benefits are additive, sub-additive (because both approaches address the same bottleneck), or super-additive (because reduced reward sparsity and reduced gradient variance reinforce each other). The specific experiment would compare four conditions on the MATH benchmark: (1) token-level PPO + terminal reward, (2) MA-PPO + terminal reward, (3) token-level PPO + process reward, (4) MA-PPO + process reward. The key question is whether combining both approaches closes more of the gap to an oracle credit assignment scheme than either alone.

Stress-testing MA-RLHF on tasks with extremely long sequences. The paper's theoretical motivation emphasizes long-sequence credit assignment, but the empirical evaluation uses datasets where the average chosen response length is relatively modest: 35 tokens for TL;DR, 83 tokens for HH-RLHF, 149 tokens for WebGPT (Table 4). These are not extremely long sequences by modern standards—multi-turn dialogue, long-form QA, document summarization, and code generation can produce outputs of 500–2000+ tokens. The paper's variance reduction argument predicts that MA-RLHF's benefits should increase with sequence length, because the variance of the token-level policy gradient grows with the number of time steps. A strong stress-test would evaluate MA-RLHF on a long-form generation task (e.g., summarization of long documents with 500+ token outputs, or multi-turn dialogue with 5+ turns) and measure whether the relative improvement over vanilla PPO grows with sequence length. The specific prediction is that the convergence speedup factor (currently 1.7–2×) should increase for longer sequences, potentially reaching 3–5× for sequences of 1000+ tokens where token-level credit assignment becomes extremely noisy. If the benefit plateaus or decreases, it would suggest that macro actions help primarily with short-to-medium-range credit assignment and that very long-range dependencies require different mechanisms.

Cross-modal and cross-architectural generalization of temporal abstraction. The paper validates MA-RLHF on autoregressive transformer language models (Gemma, Llama-3.2) for text generation tasks. The temporal abstraction principle—reduce the effective decision horizon to reduce policy gradient variance—should apply to any sequential decision-making domain where actions have local coherence and rewards are sparse relative to the sequence length. Obvious extensions include: (1) code generation with different programming languages (the APPS results use Python only; would parsing-based macro actions using language-specific AST parsers show similar benefits for Java, C++, or SQL?), (2) speech or music generation where the primitive actions are audio tokens and macro actions could correspond to phonemes, words, or musical phrases, (3) vision-language tasks where the model generates sequences of image patches and text tokens interleaved, (4) diffusion models for text-to-image generation where the denoising trajectory is a sequence of actions. For each domain, the specific question is whether domain-appropriate macro-action boundaries (e.g., beat-aligned segments for music, object-level patches for images) provide benefits beyond content-agnostic fixed-length chunking, as the paper shows for parsing-based termination on code and dialogue. A strong follow-up would identify one non-text domain, implement a domain-specific termination strategy analogous to the paper's parsing-based approach, and measure both convergence speed and final quality against a fixed n-gram baseline and a vanilla token-level baseline.

Practical Applications and Downstream Use Cases

Cost-efficient RLHF training for production language models. The most direct practical application is reducing the computational cost of the RLHF stage in production LLM training pipelines. The paper demonstrates that MA-PPO reaches a given RM score (a proxy for model quality) in approximately half the training steps of vanilla PPO on TL;DR (Figure 2, left: 1.7k vs. 3.7k steps for Gemma-2B), with similar speedups on HH-RLHF (Figure 13: 2.6k vs. 5.1k steps for 2B). For organizations training large models where each RLHF training step involves generating responses from the policy, scoring them with the reward model and critic, and performing gradient updates across billions of parameters, halving the number of required steps translates directly to reduced GPU-hours and faster iteration cycles. The paper's demonstration that the benefit persists at 27B scale (Figure 9)—and the single validation on Llama-3.2-3B (Appendix C.4)—suggests applicability beyond the specific Gemma architecture. A team training a 70B-class model could adopt MA-PPO with the default fixed 5-gram termination and expect to reduce their RLHF training budget by approximately 40–50% (accounting for the 1.7–2× step reduction, minus any minor overhead from macro-action boundary computation), with the added benefit of higher final quality (+30% RM score on TL;DR for 7B, Table 2) at the same step count if training continues to convergence.

Improved alignment quality with no inference-time cost. MA-RLHF modifies only the training procedure; the resulting checkpoint is a standard autoregressive language model with no additional inference latency, memory overhead, or architectural complexity. This makes it a "free" quality improvement from the deployment perspective—unlike methods that improve generation quality through test-time computation (e.g., best-of-N sampling, beam search, or verifier-guided decoding), MA-RLHF's benefits are baked into the model weights. The paper shows that MA-PPO outputs are preferred by human evaluators at rates of 52–74% (Figure 4) and achieve higher RM scores by 18–68% (Table 2) across tasks, without requiring any change to the serving infrastructure. The robustness to sampling temperature (Figure 21: MA-PPO maintains performance at T=1.0 where vanilla PPO degrades) is an additional practical advantage—it means the model can be deployed with higher temperature for more diverse outputs without sacrificing quality, or with lower temperature for more deterministic outputs without leaving performance on the table. For a production chatbot or API service, this combination of improved quality, zero inference overhead, and reduced sensitivity to decoding hyperparameters is a strong value proposition.

Data generation for self-improvement and distillation pipelines. When using RLHF-trained models to generate synthetic training data for further fine-tuning (as in rejection sampling fine-tuning, STaR, or ReST-style self-improvement loops), the quality of the generated data directly determines the ceiling on downstream model improvement. MA-RLHF's improvements in both mean quality (higher RM scores) and tail quality (reduced prevalence of low-scoring outputs, Figure 10) translate to cleaner training data with fewer low-quality examples that could degrade the student model. The combination of MA-RLHF with best-of-N rejection sampling (Figure 8, right) achieves RM scores of ~1.85 at N=32, T=1.2 on TL;DR—substantially higher than vanilla PPO with best-of-N (~1.35) or SFT with best-of-N (~1.15). For a self-improvement pipeline where the model generates candidate responses, scores them, and trains on the best ones, using MA-PPO as the generator would produce a higher-quality training set at any given sampling budget N, potentially accelerating the self-improvement loop. The temperature robustness (Figure 21) is particularly valuable here: data generation pipelines often use high temperature to ensure diversity, and MA-PPO's maintained quality at T=1.0 means diversity doesn't come at the cost of individual sample quality.

Domain-specific tuning of the temporal abstraction granularity for specialized applications. The paper's finding that optimal macro-action granularity varies by task (n=∞ for summarization, n=10 for dialogue, parsing-based for code) provides a recipe for practitioners deploying RLHF in specialized domains. For a legal document summarization service (structurally similar to TL;DR), using n=∞ or a large fixed n-gram would likely capture the holistic evaluation nature of summarization quality. For a customer support chatbot (structurally similar to HH-RLHF dialogue), n=10 or parsing-based termination would better capture turn-level and sub-turn structure. For a code generation assistant, using a language-specific parser to define macro actions along AST boundaries would align credit assignment with program structure. The paper doesn't provide a predictive model for choosing n on a new task, but the empirical pattern—coarser granularity for holistic evaluation tasks, moderate granularity for structured interaction tasks, syntactic granularity for formal language tasks—is interpretable and actionable. A practitioner can make an educated guess based on their task's structural similarity to one of the four tested domains, run a small pilot sweep of n ∈ {3, 5, 10, ∞} if budget allows, and default to fixed 5-gram otherwise with confidence that some improvement over vanilla PPO is virtually guaranteed (all tested strategies outperformed the baseline on all tasks).

When to Prefer This Method

The paper positions MA-RLHF against three named alternatives—vanilla token-level PPO, sequence-level REINFORCE/RLOO, and offline DPO—with explicit empirical comparisons that define clear trade-offs.

Prefer MA-RLHF (with fixed 5-gram default) when:

• You are already using token-level PPO for RLHF and want faster convergence and higher final quality with no architectural changes (+30–68% RM score improvement on summarization, +18% on dialogue, +3–8% on QA; 1.7–2× training step reduction; zero inference overhead).
• Your task requires online exploration—DPO substantially underperforms on TL;DR (RM score 0.03 vs. 1.40 for MA-PPO with n=5; Table 7), and the paper's evidence suggests offline preference optimization cannot recover the benefits of online policy-gradient-based learning for summarization and dialogue tasks.
• Your model has non-trivial base capability on the task (pass@1 well above zero). MA-RLHF amplifies existing capability; it does not create capability where none exists. On APPS Competition problems with CodeGemma-2B (pass@1 = 0.92), MA-PPO shows only +2% improvement and pass@5 actually regresses (-6%), whereas the 7B model with higher base pass@1 shows +65% improvement (Table 3).
• You are willing to accept a small hyperparameter choice (n-gram length or termination strategy) in exchange for substantial training efficiency gains, and you can use fixed 5-gram as a safe default if task-specific tuning is infeasible.

Prefer vanilla token-level PPO when:

• You are operating at the extreme frontier of your model's capability where pass@1 is near zero for most instances—macro actions reduce gradient variance but cannot create learning signal from noise, and the computational overhead of the termination strategy (even if small) provides no benefit.
• You have strong per-step or per-span reward signals already (e.g., process reward models, step-level human feedback). The paper does not test MA-RLHF with fine-grained rewards, and it is possible that when the reward signal is already dense, the marginal benefit of temporal abstraction is reduced. Until this interaction is empirically characterized, token-level PPO with dense rewards is the safer choice.
• Your sequences are very short (e.g., <20 tokens on average) where the credit assignment problem is minimal and the macro-action segmentation granularity approaches the token level regardless of the chosen n.

Prefer RLOO/REINFORCE/GRPO when:

• Your task has strong holistic evaluation structure (the quality of the entire output matters more than the quality of any sub-part) AND you value implementation simplicity over maximal performance. RLOO's RM score of 0.81 on TL;DR is close to vanilla PPO's 0.83 and far below MA-PPO's 1.40, but RLOO is simpler to implement and does not require a critic model. If 2% of RM score is acceptable in exchange for reduced system complexity, RLOO is a reasonable choice—but MA-RLHF with n=∞ would likely match or exceed RLOO's performance while keeping the same simplicity, since n=∞ MA-PPO is equivalent to a REINFORCE-style objective.

Prefer DPO when:

• You cannot afford online RL training (no reward model, no generation during training, no critic) and your task is more similar to HH-RLHF dialogue (where DPO achieves RM score 0.64 vs. MA-PPO's 1.55—a gap, but DPO is at least in positive territory) than to TL;DR summarization (where DPO achieves 0.03 vs. MA-PPO's 1.40—a 46× gap that renders DPO essentially ineffective). The paper's evidence suggests DPO's viability is highly task-dependent, and practitioners should validate on their specific domain before committing to an offline-only approach.