Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

🎯 Pitch

LLMs don't reason in a flat sequence—they follow a hidden 'preplan-and-anchor' rhythm where pivotal tokens emerge just before semantic shifts, then orchestrate all downstream logic. This paper not only reveals that structure by reading attention patterns but weaponizes it: by shifting RL credit toward those critical nodes, they boost math reasoning performance by over 10 absolute points on challenging tasks like Countdown—without changing the model architecture at all.

1. Executive Summary

This paper introduces attention dynamics as a principled lens for revealing the internal reasoning structure of LLMs during reinforcement learning, and proposes three structure-aware policy optimization strategies that perform targeted credit assignment to structurally decisive tokens. Analyzing Qwen3-4B-Base on GSM8K math prompts, the authors distinguish locally focused attention heads—which exhibit a near-diagonal sawtooth pattern tracking phrasal chunks—from globally focused heads that expose semantic anchor tokens with broad downstream influence, formalizing these observations through two metrics: Windowed Average Attention Distance (WAAD), which measures how far a token consults backward context within a clipped window, and Future Attention Influence (FAI), which quantifies a token's global importance as the average attention it receives from subsequent positions. The joint dynamics of these signals reveal a recurring preplan-and-anchor rhythm: at semantic boundaries, the model performs long-range contextual reference to generate an introductory preplan token (WAAD peaks coinciding with high token entropy), followed by or coinciding with a high-FAI anchor token that organizes downstream reasoning. Leveraging these insights, the authors design three RL credit-assignment strategies—local-chunk credit that amplifies advantages at WAAD-detected preplan tokens, global-anchor credit that amplifies high-FAI anchor tokens, and coupled rhythm credit that back-allocates credit from locally dominated anchors to their preceding preplans—and demonstrate consistent gains over standard GRPO across reasoning benchmarks, with coupled rhythm credit achieving the strongest improvements (e.g., +10.5 points on Countdown over GRPO's 52.6%, and +2.1 points averaged across mathematical reasoning benchmarks for Qwen3-4B-Base), establishing that attention-derived signals can guide effective fine-grained credit assignment without requiring any architectural modifications to the underlying RL framework.

2. Context and Motivation

The Core Problem: We Optimize LLM Reasoning Blindly

The fundamental problem this paper addresses is a mismatch between how LLMs appear to structure their reasoning internally and how we optimize them through reinforcement learning. Modern Large Reasoning Models (LRMs)—trained via Reinforcement Learning with Verifiable Rewards (RLVR)—can solve complex mathematical problems, write sophisticated code, and perform multi-step agentic decision-making. Yet the optimization process that produces these capabilities operates in the dark: sequence-level rewards are spread uniformly across all tokens in a generation, treating every token as equally responsible for the final outcome.

This uniform credit assignment is problematic because not all tokens are equally consequential for reasoning. Consider a mathematical proof: the step where the model defines a key variable or commits to a particular solution strategy is vastly more important than the tokens that merely format the response or complete habitual phrases like "by the way" or "therefore, we can see that." When RL spreads credit evenly, pivotal strategic decisions receive the same reinforcement signal as routine syntactic completion, which the paper argues limits data efficiency (the model needs more training samples to isolate which behaviors matter), interpretability (we cannot see which reasoning steps the optimization process is actually rewarding), and the reliability of performance gains (the optimization signal is diluted across many irrelevant tokens).

The consequence is that while RLVR produces models with impressive reasoning capabilities—as demonstrated by systems like DeepSeek-R1 and Kimi k1.5—the training process itself treats the model as a black box. We observe output improvements without understanding which specific reasoning behaviors the optimization is reinforcing, making it difficult to diagnose failures, improve data efficiency, or design more targeted training strategies.

Why This Matters: Beyond Opaque Optimization

This gap has both practical and scientific significance:

Practical significance: data efficiency and training stability. Training LRMs via RLVR is computationally expensive, requiring millions of sampled trajectories and thousands of GPU-hours. If we could identify which tokens in a generation are structurally decisive for correct reasoning, we could concentrate the RL optimization signal on those positions, potentially achieving the same performance with less data, or better performance with the same data. This directly impacts the cost and feasibility of developing reasoning models at scale. The paper's approach is "plug-and-play with standard RLVR" (Section 6), meaning it does not require architectural changes to existing training pipelines—it only modifies how advantages are computed per-token, making it directly applicable to production systems.

Scientific significance: bridging the interpretability-optimization gap. A substantial body of work in mechanistic interpretability has analyzed how LLMs represent and execute reasoning internally, identifying phenomena like iteration heads (Cabannes et al., 2024), semantic induction heads (Ren et al., 2024), deduction circuits (Dutta et al., 2024; Ameisen et al., 2025; Lindsey et al., 2025), and attention sinks (Xiao et al., 2023). These studies provide rich descriptive accounts of internal model behavior but largely stop at description—they rarely yield prescriptive training recipes that improve model performance. As the paper notes in Section 2:

"These accounts, however, are often descriptive and rarely yield process-aware training recipes."

The paper's ambition is to close this loop: use interpretability insights (attention dynamics that reveal reasoning structure) to directly inform the optimization process (token-level credit assignment). If successful, this transforms interpretability from a diagnostic tool into an actionable component of the training pipeline, moving beyond "understanding what the model does" to "using that understanding to make the model better."

Prior Approaches and Their Shortcomings

The paper identifies several lines of prior work that attempt to address the blind optimization problem, each with specific limitations:

1. Uniform credit assignment in standard RLVR (the dominant approach). The most widely-used RL algorithms for LLM reasoning—PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024)—spread sequence-level rewards across all tokens. In PPO, the advantage At is estimated via Generalized Advantage Estimation (GAE) and applied uniformly to each token. In GRPO, the group-normalized advantage from Equation 6 in the paper is a single scalar per sequence that is identically applied across tokens. This means that a correct solution reinforces every token in the generation equally, even those that were irrelevant to the reasoning or even potentially harmful patterns that happened to co-occur with correct outputs.

The key flaw: there is no mechanism to distinguish between tokens that were causally necessary for the correct answer and tokens that were merely correlated with it. This makes the optimization signal noisy and inefficient, particularly on complex reasoning tasks where the ratio of "structurally important" to "routine" tokens may be small.

2. Entropy-based and exploration-focused approaches. A growing body of work has identified that high-entropy tokens—positions where the model's predictive distribution is uncertain—are important for reasoning because they represent decision points where the model could take different paths (Wang et al., 2025b; Cheng et al., 2025; Cui et al., 2025). The intuition is that these "forking" tokens govern exploration: by identifying and amplifying them during RL, we encourage the model to explore diverse reasoning strategies. The paper's experiments include an entropy-based baseline that amplifies advantages for tokens with high predictive entropy.

The key flaw: entropy is a symptom, not a cause. High entropy indicates that the model is uncertain, but it does not explain why it is uncertain or what role the token plays in the reasoning structure. A token can have high entropy because it is a genuine decision point (e.g., choosing between two mathematical operations) or because it is an arbitrary stylistic choice (e.g., choosing between synonyms). Conversely, some structurally decisive tokens may have relatively low entropy if the model has already committed to a particular reasoning path by the time it reaches them. Entropy-based approaches conflate these cases.

3. Mechanistic interpretability without optimization integration. The literature contains extensive analyses of how attention heads, MLP layers, and residual streams contribute to reasoning. For example, Bogdan et al. (2025) identified "thought anchors"—LLM reasoning steps that matter for downstream inference—using receiver head analysis, and Men et al. (2024) studied look-ahead planning in attention patterns. These studies demonstrate that attention dynamics contain rich information about reasoning structure.

The key flaw: these analyses are descriptive, not prescriptive. They tell us that certain tokens or attention patterns are important, but they do not provide a mechanism for using this information during training to improve the model. The paper explicitly positions itself as bridging this gap: it uses the descriptive insights from attention analysis to design prescriptive interventions (targeted credit assignment) within the RL training loop.

4. Token-level credit assignment without structural grounding. Some prior work has attempted token-level credit assignment in RL for LLMs, such as Lin et al. (2024)'s "critical tokens" approach and Vassoyan et al. (2025)'s exploration on critical tokens. These methods identify important tokens based on external signals (e.g., contrastive estimation) or heuristics, but they do not ground the identification in the model's own internal reasoning structure.

The key flaw: without grounding in the model's own attention dynamics, token importance is defined by proxy signals that may not reflect the model's actual reasoning organization. The model may treat certain tokens as important for reasons that are opaque to external metrics but visible in its attention patterns.

How This Paper Positions Itself

The paper positions itself at the intersection of two previously disconnected research threads: attention-based mechanistic interpretability and practical RL optimization for reasoning. Its central claim is that attention dynamics provide a privileged substrate for understanding and directing reasoning because they directly reflect which tokens the model itself treats as structurally important, rather than relying on external proxies.

This positioning is concretely expressed through three design choices that distinguish it from prior work:

First, the paper derives importance signals from the model's own attention, not from external criteria. The WAAD and FAI metrics are computed from the attention maps of a single auxiliary forward pass over the generated response. This means the identification of preplan and anchor tokens is model-intrinsic: it reflects which positions the model's attention mechanism actually treats as structurally important, not which positions a human annotator or heuristic considers important. This is a fundamentally different source of signal than entropy-based approaches (which use the model's output distribution) or content-based approaches (which analyze the semantic content of tokens).

Second, the paper quantifies a specific, recurring reasoning rhythm rather than treating tokens in isolation. The preplan-and-anchor mechanism—where WAAD peaks (long-range contextual retrieval) precede or coincide with FAI peaks (tokens with broad downstream influence)—is presented as a joint dynamic that characterizes how the model transitions between reasoning chunks. This moves beyond identifying individual "important tokens" to characterizing the temporal structure of reasoning itself. The three coupling patterns documented in Section 4.2 (WAAD peaks maintain higher entropy; receiver heads and global heads surface shared anchors; FAI peaks follow or coincide with WAAD peaks) provide a multidimensional characterization of this rhythm that is empirically validated through quantitative lift over random baselines (Table 2, with lifts ranging from +42.47% to +171.49%).

Third, the paper translates descriptive insights into prescriptive interventions that are compatible with existing RL frameworks. Rather than proposing a new RL algorithm, the paper modifies only the per-token advantage scaling coefficient γt within standard PPO/GRPO objectives (Equation 12). This means the method is "plug-and-play": it can be integrated into existing RLVR pipelines by simply replacing the uniform advantage At with the shaped advantage Atγt, without changing the optimizer, the reward model, the training infrastructure, or the model architecture. The attention computation is handled by a separate auxiliary model (actor_attn) that performs a single additional forward pass, introducing minimal overhead (Section 5.1).

What the paper does NOT claim or do. The paper does not propose a new attention mechanism, a new model architecture, or a new RL objective. It does not claim that attention is the only substrate for understanding reasoning structure, or that WAAD and FAI capture all aspects of reasoning. It does not require ground-truth labels for the reasoning steps—all metrics are derived from attention maps alone. And crucially, the paper empirically validates that the attention-derived signals are causal (not merely correlational) through perturbation experiments (Table 1, Section 4.1) showing that changing high-FAI tokens substantially alters downstream reasoning trajectories while changing low-FAI tokens does not.

The paper's intellectual contribution is thus primarily methodological and empirical: it provides (1) a principled framework for extracting reasoning structure from attention dynamics, (2) quantitative evidence that this structure reflects causally important decision points, and (3) a practical recipe for using this structure to improve RL optimization efficiency, with consistent empirical gains demonstrated across model scales (4B and 8B), context lengths (1K and 8K), and reasoning task types (puzzles, QA, mathematical reasoning).

3. Technical Approach

3.1 Reader Orientation

The paper builds a structure-aware credit assignment system that modifies how reinforcement learning distributes reward signals across tokens in LLM reasoning traces. The core problem is that standard RL treats every generated token as equally responsible for the final answer, which dilutes the optimization signal — the solution is to use the model's own attention patterns to identify which tokens are structurally decisive for reasoning and amplify the RL advantage at those positions, without changing the underlying model architecture or RL algorithm.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that operate within a standard RL training loop:

Actor Infer — a vLLM-deployed instance of the policy model that generates responses (reasoning traces) to input prompts via autoregressive decoding at training temperature (T=1.0).
Actor Attn — a dedicated auxiliary model (identical weights to the policy model, implemented with standard non-flash Transformer) that performs a single additional forward pass over each generated response to extract full attention maps from selected layers, enabling computation of WAAD and FAI metrics.
Attention Signal Extractor — a post-processing module that classifies attention heads into local/global sets, aggregates their attention maps, and computes per-token WAAD and FAI values from the actor_attn output.
Credit Assignment Module — a computation layer that takes the per-token WAAD and FAI sequences, identifies preplan tokens (via WAAD drops), anchor tokens (via top-quantile FAI), and locally dominated anchors (via joint WAAD-FAI criteria), then produces per-token scaling coefficients γt according to one of three strategies.
Actor Train — a Megatron-implemented training instance that receives the prompt-response-reward-γt tuples, computes policy gradients using the standard clipped PPO/GRPO objective but with shaped advantages Atγt instead of uniform At, and updates model parameters.

Information flows in a closed loop: actor_infer generates responses → reward model/verifier scores them → actor_attn extracts attention maps for each generated response → credit assignment module computes γt per token → actor_train performs policy update with shaped advantages → updated weights are synchronized to both actor_infer and actor_attn → next iteration begins.

3.3 Roadmap for the Deep Dive

First, the head classification and aggregation procedure (Equation 7, Equation 8), which is the foundation for all subsequent metrics — understanding how the model partitions its attention into local and global regimes is prerequisite for interpreting WAAD and FAI.
Second, the two core metrics — Windowed Average Attention Distance (Definition 1) and Future Attention Influence (Definition 2) — which operationalize the local sawtooth pattern and the global anchor pattern, respectively. These are the quantitative lenses through which the preplan-and-anchor rhythm becomes visible.
Third, the empirical validation of the preplan-and-anchor rhythm through three coupling patterns (entropy-WAAD coupling, receiver-global FAI concordance, WAAD-FAI temporal alignment) and the perturbation experiments that establish the causal role of high-FAI tokens.
Fourth, the attention computation infrastructure within the RL framework (Section 5.1), which explains how attention maps are obtained during distributed training without disrupting the efficiency of flash attention — a crucial practical detail.
Fifth, the three credit assignment strategies (local-chunk, global-anchor, coupled rhythm) that translate the attention-derived signals into per-token advantage scaling coefficients γt within the standard PPO objective (Equation 12).
Sixth, the detailed experimental configuration, including all hyperparameters, model configurations, and implementation choices that make the system work end-to-end.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that attention dynamics reveal an intrinsic reasoning rhythm in LLMs, and that using this rhythm to perform targeted credit assignment in RL yields consistent performance improvements. The paper does not propose a new RL algorithm or attention mechanism; instead, it provides a methodology for extracting structurally meaningful signals from existing attention maps and a recipe for integrating those signals into standard RL training loops.

Head Grouping by Span: Separating Local from Global Attention

The first technical step is to classify each attention head in the model as either locally focused (primarily attending to nearby tokens) or globally focused (frequently attending to distant tokens). This classification is necessary because local and global heads reveal fundamentally different aspects of reasoning structure — local heads track phrasal chunking, while global heads identify semantic anchors — and aggregating them separately prevents these signals from being averaged away.

Head-level attention distance. For each attention head $(l, h)$ , where $l$ indexes the layer and $h$ indexes the head within that layer, the paper computes a scalar $d^{(l,h)}$ that measures the head's effective receptive span — the average distance it looks backward when generating response tokens:

$d^{(l,h)} = \frac{1}{|R|} \sum_{t \in R} \sum_{s=1}^{t} A^{(l,h)}_{t,s} (t - s)$

where $R$ is the set of response token positions (excluding the prompt), $A^{(l,h)}_{t,s}$ is the attention weight from position $t$ to position $s$ in head $(l,h)$ , and $(t - s)$ is the positional distance between the query token and the attended token.

What it computes: For each response token $t$ , the inner sum computes a weighted average of the distances to all previously attended positions, with the weights being the attention probabilities (since $\sum_{s=1}^{t} A^{(l,h)}_{t,s} = 1$ ). This gives the expected distance that token $t$ reaches backward in head $(l,h)$ . Averaging this expected distance over all response tokens yields the head-level metric $d^{(l,h)}$ . A small value (e.g., 2-5 tokens) means the head concentrates its attention very close to the diagonal — it primarily looks at the immediately preceding words. A large value (e.g., 20-50 tokens) means the head frequently reaches far back into the context.

Why this form: Using attention-weighted mean distance rather than, say, the maximum attention position or a binary threshold on distance captures the full distribution of the head's attention behavior. A head might attend to both nearby and distant tokens, and the weighted average reflects the overall tendency. The alternative — simply checking whether attention exceeds some distance threshold — would be sensitive to the choice of threshold and would not distinguish between a head that occasionally looks far back versus one that consistently does so.

Head classification by quantile. After computing $d^{(l,h)}$ for all $H \times L$ heads (where $H$ is heads per layer and $L$ is layers), the paper sorts heads by this distance metric and designates the lowest quantile (e.g., bottom 30%) as the local-focused set $\mathcal{H}_{\text{loc}}$ and the highest quantile (e.g., top 30%) as the global-focused set $\mathcal{H}_{\text{glob}}$ . The middle 40% of heads are excluded from the analysis, which is a deliberate design choice: these heads exhibit intermediate behavior that does not clearly showcase either the local sawtooth or global anchor pattern, and including them would dilute the signal in the aggregated maps.

Aggregating head sets. Once the head sets are defined, the paper aggregates the attention maps within each set by simple averaging:

$\bar{A}^{\text{loc}} = \frac{1}{|\mathcal{H}_{\text{loc}}|} \sum_{(l,h) \in \mathcal{H}_{\text{loc}}} A^{(l,h)}, \quad \bar{A}^{\text{glob}} = \frac{1}{|\mathcal{H}_{\text{glob}}|} \sum_{(l,h) \in \mathcal{H}_{\text{glob}}} A^{(l,h)}$

where $|\mathcal{H}_{\text{loc}}|$ and $|\mathcal{H}_{\text{glob}}|$ are the number of heads in each set.

What it computes: $\bar{A}^{\text{loc}}$ is a single $N \times N$ lower-triangular matrix (where $N$ is the total sequence length including prompt and response) that represents the average local-head attention pattern, and $\bar{A}^{\text{glob}}$ is the analogous average for global heads. These aggregated maps are the inputs to all subsequent metric computations. Averaging across heads within each class amplifies the consistent structural patterns (the sawtooth in local heads, the vertical stripes in global heads) while suppressing head-specific noise.

Why aggregate rather than analyze individual heads: Individual attention heads can be noisy, and the characteristic patterns (sawtooth, vertical stripes) are distributed across multiple heads rather than being perfectly expressed in any single head. Aggregation produces stable, interpretable patterns that the WAAD and FAI metrics can operate on reliably. The choice to use only the bottom/top 30% quantiles rather than a hard distance threshold acknowledges that "local" and "global" are relative to the specific model and sequence — the quantile approach adapts automatically without requiring manual threshold tuning.

Layer selection for attention extraction. When computing attention maps in the RL training pipeline (Section 5.1), the paper does not extract maps from all layers. Instead, it samples from five evenly spaced layers within the middle third of the network — specifically, from layers $\lfloor L/3 \rfloor$ to $\lfloor 2L/3 \rfloor$ , where $L$ is the total number of layers. This choice is motivated by the observation that early layers primarily handle low-level feature extraction and late layers primarily prepare the output distribution, while the middle layers are where high-level semantic and structural processing occurs, making them most informative for capturing reasoning rhythms. Sampling five layers provides a representative snapshot without the memory and compute overhead of storing attention from all layers.

Windowed Average Attention Distance (WAAD): Quantifying the Local Sawtooth

Once the local-focused aggregate attention map $\bar{A}^{\text{loc}}$ is computed, the paper needs a per-token metric that captures the sawtooth pattern — the alternation between tightly local attention within phrasal chunks and longer-range attention at chunk boundaries. The WAAD metric is designed to make this alternation visible as a one-dimensional signal over the token sequence.

Definition and computation. For each response position $t \in R$ , with a clipping window size $W$ :

$\text{WAAD}_t = \sum_{s=1}^{t} \bar{A}^{\text{loc}}_{t,s} \min(t - s, W)$

where $\bar{A}^{\text{loc}}_{t,s}$ is the aggregated local-head attention weight from token $t$ to token $s$ , $t - s$ is the actual backward distance, and $W$ is the window clipping parameter (set to $W = 10$ in all experiments).

What it computes: WAADt measures how far token $t$ reaches backward within the local attention regime, but with a crucial modification — distances beyond $W$ are clipped to $W$ . Concretely, for each past position $s$ , the contribution to WAADt is the product of (a) how much attention token $t$ pays to $s$ and (b) the distance to $s$ , but capped at $W$ . The sum over all $s \leq t$ yields the expected clipped backward distance for token $t$ .

Why clipping is necessary: Without clipping (i.e., using the raw distance $t - s$ ), the WAAD signal would be dominated by absolute positional effects — tokens later in the sequence would naturally have larger raw distances simply because they have more preceding tokens to attend to, regardless of whether they are actually consulting long-range context. The clipping window $W=10$ creates a local regime: within the window, WAAD distinguishes between very local attention (distance 1-3 tokens, typical of within-chunk processing) and moderately longer attention (distance 7-10 tokens, typical of chunk boundaries where the model reaches back to the beginning of the current reasoning segment). Beyond 10 tokens, all distances are treated identically, preventing positional artifacts from obscuring the chunk-boundary signal.

Why $W=10$ specifically: The window size reflects the approximate length of a typical phrasal or semantic chunk in the analyzed reasoning traces. Within a chunk of 5-10 tokens, attention is expected to stay highly local (distance 1-3). When a new chunk begins, the model often reaches back 5-10 tokens to the beginning of the previous chunk or to a key earlier reference point. Setting $W$ to 10 captures this range without being dominated by the overall sequence length.

Relationship to the sawtooth pattern. When visualized as a sequence over tokens, WAADt produces a characteristic peak-valley alternation. Peaks (high WAAD values) correspond to chunk-onset tokens where the model consults longer-range context — these are the "preplan" tokens that set up the next reasoning step. Valleys (low WAAD values) correspond to within-chunk tokens where attention remains tightly local — these are routine continuations that follow naturally from the immediately preceding context. The transition between a peak and the subsequent valley (i.e., a sharp drop in WAAD over consecutive tokens) indicates a chunk boundary where the model has finished consulting context and settled into local continuation mode. This drop is quantified as the absolute difference between consecutive WAAD values:

$\Delta_t = |\text{WAAD}_t - \text{WAAD}_{t+1}|$

Large $\Delta_t$ identifies the token at the peak of the sawtooth — the position just before attention collapses back to local mode — which the paper uses as the preplan token selection signal.

Design rationale — why attention-weighted distance rather than attention entropy or other statistics: The attention-weighted distance directly captures the spatial extent of context consultation, which is the specific behavior the paper hypothesizes distinguishes chunk boundaries from within-chunk tokens. Alternative metrics like attention entropy (which measures how uniformly attention is distributed across past positions) would conflate two different cases: a token might have high entropy because it attends uniformly to many nearby positions (still within-chunk behavior) or because it reaches far back selectively (chunk-boundary behavior). WAAD cleanly separates these because the distance weighting penalizes near-diagonal attention and rewards longer backward reaches.

Future Attention Influence (FAI): Quantifying the Global Anchor Pattern

The complementary global perspective asks: which tokens receive sustained attention from many future positions, indicating they serve as persistent reference points for downstream reasoning? The FAI metric quantifies this by averaging the attention that a token receives from subsequent positions within a controlled horizon.

Definition and computation. For a token at position $s$ , with lower horizon $H_{\text{lo}}$ and upper horizon $H_{\text{hi}}$ :

$\text{FAI}_s = \frac{1}{|T(s)|} \sum_{t \in T(s)} \bar{A}^{\text{glob}}_{t,s}$

where $T(s) = \{ t \mid t \in R, s + H_{\text{lo}} \leq t \leq \min(N, s + H_{\text{hi}}) \}$ is the set of future response positions that fall within the horizon window from $s$ , and $\bar{A}^{\text{glob}}_{t,s}$ is the aggregated global-head attention weight from future token $t$ back to token $s$ .

What it computes: FAI_s is the average attention that token $s$ receives from tokens in the window $[s + H_{\text{lo}}, s + H_{\text{hi}}]$ (capped by sequence length $N$ ). The lower horizon $H_{\text{lo}}$ prevents immediate neighbors from dominating the metric — a token might receive high attention from the very next token simply because they are part of the same phrase, which does not indicate long-range anchoring. The upper horizon $H_{\text{hi}}$ controls the range over which influence is measured, preventing the metric from being diluted by tokens very far in the future whose attention patterns may be driven by unrelated context.

Why the horizon window is necessary: Without the lower horizon, FAI would be inflated by tokens that are part of multi-token words or tightly bound phrases — for example, the first token of "calculation" might receive high attention from the second token of "calculation" because they form a single tokenized unit, but this is not semantically meaningful anchoring. The lower horizon $H_{\text{lo}}$ (set via the range $H \in [10, 50]$ in the paper, indicating a window starting 10 tokens after $s$ ) excludes these immediately adjacent positions. Without the upper horizon, FAI would measure average attention over the entire remainder of the sequence, which would be dominated by tokens far from $s$ that may attend to $s$ only weakly and variably. The upper horizon $H_{\text{hi}}$ (50 tokens after $s$ ) focuses the measurement on the region where anchoring effects are most pronounced — the next few reasoning steps that build on the information at $s$ .

What high FAI indicates. Tokens with high FAI values are those that future tokens consistently attend to across the 10–50 token horizon. These are typically semantic anchors — key definitions, intermediate results, decision points, or conclusions that subsequent reasoning steps repeatedly reference. For example, in a math solution, a token like "x=5" or "therefore" might receive high FAI because the model continues to look back at that intermediate result as it performs subsequent calculations. Visually, high-FAI tokens correspond to vertical stripes in the global attention map — columns where the attention weight is elevated across many future rows.

Why use only global heads for FAI: The paper restricts FAI computation to $\bar{A}^{\text{glob}}$ rather than using all attention heads because global heads are precisely the ones that exhibit long-range attention behavior. If local heads were included, their near-diagonal attention patterns would contribute noise — local heads pay little attention to distant past tokens, so their contribution to FAI for early tokens would be near zero, diluting the signal. Restricting to global heads ensures FAI measures only the head-specific attention patterns that actually perform long-range referencing.

Receiver head concordance (Coupling Pattern 2). The paper independently validates FAI by comparing global-head FAI with receiver-head FAI, following the receiver head definition from Bogdan et al. (2025). Receiver heads are identified by the kurtosis (peakedness) of their column-wise attention distribution — heads where a few tokens receive disproportionately high attention from all future positions. The paper finds that FAI computed from receiver heads is strongly correlated with FAI computed from global-focused heads (Table 2 reports 60.84% co-occurrence of FAI peaks vs. 22.41% under random chance, a +171.49% lift). This is significant because it shows that two independent head-filtering criteria — span-based (distance-based quantile) and kurtosis-based — converge on identifying the same anchor tokens, providing convergent validation that the anchor phenomenon is robust and not an artifact of a particular head selection method.

Perturbation validation. To establish that FAI reflects causality rather than mere correlation, the paper conducts perturbation experiments (Section 4.1): at selected positions, it forces the model to generate an alternative token (from the top-k highest-logit candidates instead of the greedy choice) and rolls out the subsequent trajectory, comparing it to the original via Jaccard similarity of content words. The finding (Table 1) that perturbations at high-FAI positions produce substantially lower Jaccard similarity (0.534) than perturbations at low-FAI positions (0.631), with 87.14% of trials showing greater deviation at high-FAI positions, provides causal evidence that high-FAI tokens govern downstream reasoning structure — changing them redirects the reasoning trajectory, whereas changing low-FAI tokens primarily affects surface phrasing without altering the overall plan. This is not a correlational observation; it is an interventionist demonstration that the anchor tokens identified by FAI are causally necessary for the specific reasoning path the model follows.

The Preplan-and-Anchor Rhythm: Joint Dynamics of WAAD and FAI

The paper's central empirical claim is that WAAD and FAI are not independent — they exhibit systematic joint dynamics that reveal a recurring preplan-and-anchor mechanism underlying LLM reasoning. Section 4.2 documents three coupling patterns, each validated quantitatively against random baselines.

Coupling Pattern 1: WAAD peaks maintain higher token entropies. At positions where WAAD spikes (indicating long-range context retrieval), the model's predictive distribution $p_\theta(\cdot \mid q, o_{<t})$ exhibits higher entropy $H_t = -\sum_v p_\theta(v \mid \cdot) \log p_\theta(v \mid \cdot)$ . Table 2 reports that the average entropy at WAAD peaks is 0.3608, compared to 0.2386 under random position selection — a +51.97% lift.

Mechanistic interpretation: When the immediate local context sufficiently determines the next token (e.g., after "by the", "way" has high probability and low entropy), attention stays near the diagonal and WAAD is low. When the model approaches a semantic boundary where the next step cannot be predicted from local cues alone, uncertainty increases (high entropy) and the model compensates by reaching further back to retrieve disambiguating information (high WAAD). In this sense, WAAD peaks and high-entropy tokens are two manifestations of the same underlying state: the model recognizes that the local context is insufficient and must engage in deliberate, long-range consultation to decide what comes next. This coupling is mechanistically important because it connects the attention-based signal (WAAD) to a commonly-used behavioral signal (entropy), showing that the preplan token identification has an observable counterpart in the model's output uncertainty.

Coupling Pattern 2: Receiver heads and global-focused heads surface shared anchors. As discussed above, FAI peaks computed from receiver heads and from global-focused heads overlap at 60.84% compared to 22.41% under random chance (Table 2), a +171.49% lift. This convergence across independent head-filtering criteria strengthens the claim that the anchor phenomenon is a genuine structural feature of the model's attention, not an artifact of the specific quantile-based head selection.

Coupling Pattern 3: FAI peaks follow or coincide with WAAD peaks. This is the central finding that defines the preplan-and-anchor rhythm. Table 2 reports that 52.53% of FAI peaks either coincide with or immediately follow a WAAD peak, compared to 36.87% under random shuffling — a +42.47% lift. While this lift is smaller than the other two couplings, it is still substantially above chance, and the paper argues it reflects a consistent two-beat temporal structure:

Preplan phase: As the model approaches a semantic boundary (the end of one reasoning chunk and the beginning of the next), WAAD spikes as the model consults longer-range context to determine what the next chunk should contain. This spike produces a token — the preplan token — that introduces or prepares the forthcoming reasoning step. The preplan token often has high entropy, reflecting the deliberative nature of this contextual consultation.
Anchor phase: At the same position (if the preplan token itself serves as the anchor) or at the immediately following position, the model emits an anchor token with high FAI — a token that subsequent tokens will repeatedly attend to as they execute the reasoning step that the preplan token introduced.

Why the anchor often follows rather than coincides with the preplan: The paper argues that to preserve fluency and conventional phrasing, the model frequently produces an introductory token first (e.g., "So," "Therefore," "Now we can") before emitting the semantically contentful anchor token (e.g., "x = 5", "the total cost is"). From a local perspective, the anchor token can be dominated by this immediately preceding introductory token — its attention may be drawn primarily to the nearby preplan token rather than reaching back on its own. This is the motivation for the coupled rhythm credit strategy (Section 5.2, strategy 3), which recognizes that the preplan token and the anchor token form a functional unit where the preplan sets up the reasoning step and the anchor crystallizes it for downstream reference.

The locally dominated anchor concept. A token is classified as a locally dominated anchor if it has high FAI (it is a global anchor) but low WAAD (its own backward attention is primarily local, indicating it is strongly licensed by its immediate predecessor rather than performing its own long-range consultation). Formally, for an anchor token $t$ in the candidate set $T_{\text{glob}}$ , it is locally dominated if:

$\text{WAAD}_t \leq \tau_{\text{waad}} \quad \text{and} \quad \max_{u \in \{t-k, \dots, t-1\}} \Delta_u \geq \tau_\Delta$

where $\tau_{\text{waad}}$ is a WAAD threshold, $\tau_\Delta$ is a threshold on the WAAD drop magnitude, and $k$ is a small neighborhood size (set to 1, 2, or 3 in experiments). This condition identifies anchors where (a) the anchor's own attention is local (low WAAD), and (b) one of the immediately preceding tokens is a WAAD peak — indicating a preplan token just emitted the anchor. The key insight is that locally dominated anchors have limited capacity for independent optimization because their generation is largely determined by the preceding preplan token; therefore, credit should be partially reallocated from these anchors to their associated preplan tokens, which is the mechanism of the coupled rhythm credit strategy.

Quantitative grounding of coupling patterns. The random baselines in Table 2 are computed by shuffling peak positions while preserving their count, then recomputing the coupling statistic under this null distribution. For example, for Coupling Pattern 3, the "random" 36.87% represents the expected proportion of FAI peaks that would coincide with or follow WAAD peaks if both sets of peaks were distributed randomly and independently across the token sequence. The observed 52.53% is 15.66 percentage points higher, which the lift calculation $(52.53 - 36.87) / 36.87 = 42.47\%$ expresses as a relative increase over chance. These lifts are substantial enough to reject the null hypothesis of independence, but the fact that all observed values are well below 100% indicates that the couplings are statistical tendencies rather than deterministic rules — there are many positions where WAAD peaks without high FAI, or high FAI without WAAD peaks, reflecting the complexity of real reasoning trajectories.

Attention Computation Infrastructure Within the RL Framework

The paper's method requires extracting full attention maps from model forward passes, but modern LLM training and inference systems use flash attention for efficiency — an algorithm that computes attention outputs without materializing the full $N \times N$ attention matrix in memory. Flash attention reduces memory complexity from $O(N^2)$ to $O(N)$ by tiling the computation, which is essential for training on long sequences but discards the attention weights needed for WAAD and FAI. The paper addresses this through a dedicated architectural component (Section 5.1).

Three-model architecture. The paper maintains three separate instances of the policy model, all with identical weights that are synchronized after each policy update:

actor_infer — deployed with vLLM (a high-throughput inference engine) for autoregressive response generation. Uses flash attention for speed, so attention maps are not accessible. Responsible for generating the prompt-response pairs that constitute the RL experience data.
actor_attn — implemented with a standard Transformer using eager attention (i.e., the full attention matrix is computed and stored during the forward pass). This model takes the concatenated prompt and generated response and performs a single forward pass, during which it samples attention maps from five evenly spaced layers in the middle third of the network (layers $\lfloor L/3 \rfloor$ to $\lfloor 2L/3 \rfloor$ ). These maps are used to compute WAAD and FAI.
actor_train — implemented in Megatron (a distributed training framework with model parallelism) for large-scale gradient computation and parameter updates. Uses flash attention for training efficiency. Receives the shaped advantages (with γt coefficients computed from actor_attn's attention maps) and performs standard PPO/GRPO policy gradient updates.

Information flow within one RL iteration. The sequence proceeds as follows:

Step 1 (inference): actor_infer receives a batch of prompts $q \sim \mathcal{Q}$ , generates responses $o \sim \pi_\theta(\cdot \mid q)$ autoregressively (thousands of forward passes, one per token, using flash attention for efficiency).
Step 2 (reward evaluation): Each response $o$ is scored by a verifier or reward model, producing scalar rewards $r(o)$ . For mathematical reasoning, this is typically a binary correctness check; for the Countdown puzzle, it is whether the expression evaluates to the target number.
Step 3 (attention extraction): For each generated response, the concatenated prompt-response sequence is fed through actor_attn in a single forward pass (not an autoregressive pass — the full sequence is already generated, so this is a standard Transformer forward pass similar to training). During this pass, the full attention maps from the selected five layers are extracted and stored. The overhead is minimal: whereas actor_infer performs thousands of forward passes for token-by-token generation, actor_attn performs exactly one additional forward pass per response.
Step 4 (credit assignment): Attention maps from actor_attn are processed to (a) classify heads into $\mathcal{H}_{\text{loc}}$ and $\mathcal{H}_{\text{glob}}$ using $d^{(l,h)}$ , (b) aggregate maps within each set to produce $\bar{A}^{\text{loc}}$ and $\bar{A}^{\text{glob}}$ , (c) compute per-token WAAD from $\bar{A}^{\text{loc}}$ and FAI from $\bar{A}^{\text{glob}}$ , (d) identify preplan tokens, anchor tokens, and locally dominated anchors according to the chosen credit assignment strategy (local-chunk, global-anchor, or coupled rhythm), and (e) compute the per-token scaling coefficients $\gamma_t$ .
Step 5 (training): The prompt-response-reward-γt tuples are fed to actor_train, which computes policy gradients using the shaped advantage $\tilde{A}_t = A_t \gamma_t$ (or $\hat{A}_{i,t} \gamma_t$ for GRPO) within the standard clipped PPO/GRPO objective. All γt values are detached from the gradient computation graph, meaning the attention analysis does not backpropagate gradients — it only scales the advantages multiplicatively.
Step 6 (synchronization): Updated weights from actor_train are copied to both actor_infer and actor_attn, ensuring all three instances remain identical. The next iteration begins from Step 1 with the updated policy.

Why actor_attn is necessary rather than modifying actor_infer to output attention maps: Modifying vLLM to output full attention maps would defeat its core optimization — flash attention's memory savings are what enable high-throughput inference on long sequences. The two-instance approach (actor_infer for generation, actor_attn for attention analysis) preserves inference speed while incurring only a small additional cost for the single analysis forward pass. The paper notes that this cost is minimal compared to the thousands of forward passes already required for autoregressive generation.

Why only five layers from the middle third: Extracting attention maps from all layers would be memory-intensive and would include layers whose attention patterns are not informative for reasoning structure — early layers focus on low-level token representations and late layers prepare the output vocabulary distribution. The middle third is where high-level semantic and structural processing occurs, making it the most relevant for identifying preplan and anchor patterns. Sampling five evenly spaced layers within this region provides a representative snapshot of the model's reasoning-relevant attention behavior without the overhead of full-layer extraction.

Why γt is applied only to nonnegative advantages: The paper specifies (Section 5.2) that "all shaping signals are detached from gradients and applied only to nonnegative advantages." This means that when the advantage $A_t$ is negative (the action was worse than expected), $\gamma_t$ is not used to amplify the penalty — the token receives the standard negative advantage regardless of whether it is a preplan or anchor token. This design choice prevents the credit assignment from amplifying penalties at structurally important positions that happened to produce a poor token variant (e.g., a preplan token that introduced a wrong approach). Amplifying penalties at these positions would be counterproductive because it would suppress the planning behavior itself rather than encouraging better plans.

Targeted Credit Assignment Strategies: Translating Attention Signals into RL Advantage Weights

The final technical contribution is a set of three strategies for converting the attention-derived signals (WAAD peaks, FAI peaks, WAAD-FAI joint dynamics) into per-token advantage scaling coefficients γt within the standard RL objective. All three strategies operate by modifying only the advantage term — the rest of the PPO/GRPO objective (clipping, importance sampling, KL penalty) remains unchanged.

General framework. The paper expresses the modified PPO objective as:

$J(\theta) = \mathbb{E}_{q \sim \mathcal{Q}, o \sim \pi_\theta(\cdot \mid q)} \left[ \frac{1}{|o|} \sum_{t=1}^{|o|} \min \left( \frac{\pi_\theta(o_t \mid q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t \mid q, o_{<t})} A_t \gamma_t, \text{clip} \left( \frac{\pi_\theta(o_t \mid q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t \mid q, o_{<t})}, 1 - \epsilon, 1 + \epsilon \right) A_t \gamma_t \right) \right]$

where $\pi_\theta$ is the current policy, $\pi_{\theta_{\text{old}}}$ is the frozen policy from the previous iteration, $A_t$ is the estimated advantage at step $t$ (computed via GAE for PPO or group normalization for GRPO), $\epsilon$ is the clipping hyperparameter, and $\gamma_t$ is the strategy-specific scaling coefficient.

What $\gamma_t$ does: The coefficient $\gamma_t$ multiplicatively scales the advantage at each token. A value of $\gamma_t = 1$ recovers the standard uniform credit assignment (no modification). A value of $\gamma_t > 1$ (e.g., $1.5$ , the amplification factor used throughout) increases the magnitude of the advantage at that position, which strengthens the gradient signal for that token in the policy update. For positive advantages (the response was better than expected), increasing $\gamma_t$ strengthens the reinforcement of the token; for negative advantages, $\gamma_t$ is not applied (as noted above), so the amplification only affects tokens in trajectories that are being positively reinforced.

Why multiplicative scaling rather than additive bonus or separate loss term: Multiplicative scaling preserves the relative structure of the advantage signal within a trajectory while selectively amplifying specific positions. An additive bonus would shift all advantages by a constant, potentially making negative advantages positive (which could reinforce bad tokens). A separate loss term would introduce a new hyperparameter governing the tradeoff between the standard RL objective and the token-level objective. Multiplicative scaling cleanly integrates into the existing objective without introducing new tradeoff hyperparameters beyond the amplification factor $\gamma_{\text{amp}}$ and the selection criteria thresholds.

Strategy 1: Local-chunk credit ( $\gamma_t^{\text{local}}$ ). This strategy amplifies advantages at preplan tokens — the introductory tokens at chunk boundaries identified by WAAD drops.

Token selection: For each response, the WAAD variation between consecutive positions is computed as $\Delta_t = |\text{WAAD}_t - \text{WAAD}_{t+1}|$ . Tokens with the largest $\Delta_t$ values correspond to positions where WAAD drops sharply from peak to valley — these are the tokens at the peak of the sawtooth, which the paper identifies as preplan tokens that introduce new reasoning chunks. The top 40% of tokens by $\Delta_t$ (i.e., $q = 0.4$ ) are selected as the preplan set $T_{\text{loc}}$ :

$T_{\text{loc}} = \text{Top}_Q(\Delta_t, q = 0.4)$

The scaling coefficient is then:

$\gamma_t = 1 + (\gamma_{\text{amp}} - 1) \cdot \mathbf{1}\{t \in T_{\text{loc}}\}$

where $\gamma_{\text{amp}} = 1.5$ and $\mathbf{1}\{t \in T_{\text{loc}}\}$ is an indicator that is 1 if $t$ is in the selected set and 0 otherwise.

What this does: For the 40% of tokens with the largest WAAD drops, the advantage is multiplied by 1.5 instead of 1.0. For the remaining 60% of tokens, the advantage is unchanged (multiplied by 1.0). This concentrates the RL optimization signal on the positions where the model transitions between reasoning chunks — the points where it performs deliberate, long-range consultation to set up the next step. The amplification is uniform across all preplan tokens (all get $\gamma_{\text{amp}} = 1.5$ ), meaning the strategy does not distinguish between preplan tokens of different importance; it simply emphasizes all chunk-boundary tokens equally.

Why $q = 0.4$ and $\gamma_{\text{amp}} = 1.5$ : These hyperparameters were swept in ablations (Section 5.3.2, Figure 7 and Table 5). The choice of $q = 0.4$ means 40% of tokens are amplified, which reflects the observation that a substantial fraction of tokens participate in reasoning chunk management. Both smaller ratios ( $q = 0.2$ amplifies too few tokens, missing important preplan signals) and larger ratios ( $q \geq 0.6$ amplifies too many tokens, diluting the signal) reduce performance. The amplification factor of $\gamma_{\text{amp}} = 1.5$ provides a moderate boost — enough to shift the optimization focus but not so large as to destabilize training by over-emphasizing a small number of positions relative to the rest of the trajectory.

Mechanistic rationale: By amplifying preplan tokens, the strategy encourages the policy to invest more capacity in the contextual consultation that precedes each new reasoning chunk. Since these tokens set the direction for the subsequent chunk, improving their generation quality has disproportionate impact on downstream correctness.

Strategy 2: Global-anchor credit ( $\gamma_t^{\text{global}}$ ). This strategy amplifies advantages at anchor tokens — the high-FAI tokens that organize downstream reasoning.

Token selection: Tokens are scored by their FAI values, and the top 40% by FAI are selected as the anchor set $T_{\text{glob}}$ :

$T_{\text{glob}} = \text{Top}_Q(\text{FAI}, q = 0.4)$

The scaling coefficient is:

$\gamma_t = 1 + (\gamma_{\text{amp}} - 1) \cdot \mathbf{1}\{t \in T_{\text{glob}}\}$

where $\gamma_{\text{amp}} = 1.5$ as before.

What this does: The 40% of tokens with the highest downstream influence (as measured by average future attention) receive a 1.5× advantage amplification. This directs the RL optimization toward positions that the model's own attention mechanism treats as persistent reference points — intermediate results, key definitions, and decision points that future tokens rely on.

Mechanistic rationale: Anchor tokens articulate and preserve core semantic commitments that the remainder of the reasoning trajectory depends on. If an anchor token is imprecise or wrong, the entire downstream reasoning chain suffers. By amplifying credit at these positions, the strategy accelerates the propagation of the final correctness signal (the reward) backward to the specific tokens that most strongly determined that outcome, making the credit assignment more targeted than uniform spreading.

Contrast with local-chunk credit: Local-chunk credit emphasizes temporal structure (which tokens initiate new reasoning segments), while global-anchor credit emphasizes causal structure (which tokens have broad downstream influence). These are not identical — some preplan tokens may have low FAI (they introduce a chunk but are not themselves persistent reference points), and some anchor tokens may not coincide with WAAD peaks (they may be within-chunk tokens that happen to be highly influential). The two strategies target different aspects of reasoning structure, and their complementary nature is demonstrated by the fact that the coupled rhythm credit (which combines both signals) outperforms either strategy individually (Table 3, Table 4).

Strategy 3: Coupled rhythm credit ( $\gamma_t^{\text{coupled}}$ ). This strategy integrates the preplan and anchor signals, and additionally back-allocates credit from locally dominated anchors to their preceding preplan tokens.

Motivation for back-allocation: As discussed in Section 4.2 (Coupling Pattern 3), many anchor tokens are locally dominated — their generation is strongly determined by the immediately preceding preplan token rather than by their own long-range context processing. A locally dominated anchor has limited capacity for independent optimization because its token choice is largely constrained by the preplan token. If RL credit is assigned entirely to the anchor position, the gradient signal pushes on a token that has little room to change independently. By back-allocating a fraction of the anchor's credit to the preplan token, the strategy pushes on the more flexible upstream position that sets the context for the anchor.

Locally dominated anchor identification: An anchor token $t \in T_{\text{glob}}$ is classified as locally dominated (denoted $t \in D$ ) if it satisfies two conditions:

$\text{WAAD}_t \leq \tau_{\text{waad}} \quad \text{and} \quad \max_{u \in \{t-k, \dots, t-1\}} \Delta_u \geq \tau_\Delta$

The first condition ( $\text{WAAD}_t \leq \tau_{\text{waad}}$ ) identifies that the anchor's own backward attention is local — it is predominantly attending to nearby tokens rather than performing long-range consultation. The second condition ( $\max_{u \in \{t-k, \dots, t-1\}} \Delta_u \geq \tau_\Delta$ ) identifies that one of the immediately preceding $k$ tokens is a WAAD peak (a preplan token), meaning the anchor is emitted just after a chunk boundary.

Credit reallocation: For each locally dominated anchor $t \in D$ , the paper identifies its associated introductory token $\text{intro}(t)$ (the preplan token that precedes it) and reallocates a fraction $\alpha \in [0, 1]$ of the amplification bonus from the anchor to the preplan. The scaling coefficient becomes:

$\gamma_t = 1 + (\gamma_{\text{amp}} - 1) \cdot \mathbf{1}\{t \in T_{\text{glob}} \setminus D\} + (1 - \alpha)(\gamma_{\text{amp}} - 1) \cdot \mathbf{1}\{t \in D\} + \alpha(\gamma_{\text{amp}} - 1) \cdot \mathbf{1}\{t \in I(D)\}$

where $I(D) = \{\text{intro}(s) : s \in D\}$ is the set of introductory preplan tokens associated with the locally dominated anchors, and $\gamma_{\text{amp}} = 1.5$ is the base amplification factor.

Breaking down the four terms:

For tokens not in $T_{\text{glob}}$ and not in $I(D)$ (the majority of tokens): $\gamma_t = 1$ , i.e., standard uniform credit.
For anchor tokens not locally dominated ( $t \in T_{\text{glob}} \setminus D$ ): $\gamma_t = 1 + (\gamma_{\text{amp}} - 1) = \gamma_{\text{amp}} = 1.5$ , i.e., full amplification.
For anchor tokens that are locally dominated ( $t \in D$ ): $\gamma_t = 1 + (1 - \alpha)(\gamma_{\text{amp}} - 1)$ . If $\alpha = 0$ , this is full amplification (1.5); if $\alpha = 0.5$ , this is half amplification (1.25); if $\alpha = 1$ , this is no amplification (1.0, all credit moved to preplan).
For preplan tokens that introduce a locally dominated anchor ( $t \in I(D)$ ): $\gamma_t = 1 + \alpha(\gamma_{\text{amp}} - 1)$ . These tokens receive the credit that was taken from the locally dominated anchor.

What $\alpha$ controls: The hyperparameter $\alpha$ (implied to be around 0.5 based on the intuition of "reallocate a fraction," though the exact value is not explicitly stated in the paper) determines how much of the anchor's amplification bonus is shifted to the preplan. At $\alpha = 0$ , no reallocation occurs (reduces to Strategy 2). At $\alpha = 1$ , locally dominated anchors receive no amplification and all their credit goes to the preplan. The intermediate value balances the intuition that both the preplan (which sets up the context) and the anchor (which crystallizes the commitment) contribute to the reasoning structure, and both should receive some reinforcement.

Why this strategy outperforms the individual strategies: The coupled rhythm credit addresses a specific limitation of Strategies 1 and 2: Strategy 1 treats all preplan tokens equally regardless of whether the chunk they introduce contains an important anchor, and Strategy 2 treats all anchor tokens equally regardless of whether they are locally dominated and thus have limited optimization capacity. The coupled strategy integrates these signals — it allocates credit to preplan tokens specifically when they precede an important anchor, and it prevents over-assigning credit to anchor positions that cannot independently respond to optimization pressure. This produces a more nuanced credit assignment that aligns with the two-beat preplan-and-anchor structure while respecting the local dominance constraint.

Implementation details common to all strategies:

Top-k ratio $q$ : All strategies use $q = 0.4$ (top 40% of tokens by the selection metric) for identifying target sets. Ablation experiments (Table 5) confirm this is optimal — $q = 0.2$ (top 20%) under-selects and $q \geq 0.6$ over-selects, both degrading performance.
Amplification factor $\gamma_{\text{amp}}$ : Fixed at 1.5 across all strategies and experiments. This factor was not extensively ablated in the paper beyond the implicit validation that the strategies work.
Neighborhood size $k$ : For locally dominated anchor detection in Strategy 3, $k \in \{1, 2, 3\}$ , indicating that the search for a preceding WAAD peak checks the immediate 1–3 tokens before the anchor.
Application to nonnegative advantages only: All $\gamma_t$ values are applied only when $A_t \geq 0$ , preventing amplification of penalties.
Gradient detachment: The $\gamma_t$ values are computed from the attention maps and treated as constants during the policy gradient computation — they are not differentiated through.

Relationship to GRPO. While the paper presents the shaped advantage in the PPO objective (Equation 12), the experiments are implemented on top of GRPO (Group Relative Policy Optimization). GRPO differs from PPO in two key ways: (1) it omits the value function (critic) entirely, and (2) advantages are estimated via group normalization rather than GAE — specifically, for a prompt $q$ with $G$ sampled responses, the group-normalized advantage for response $i$ at step $t$ is:

$\hat{A}_{i,t} = \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)}$

This advantage is a single scalar per response, identically applied across all tokens. The paper's method replaces this with $\hat{A}_{i,t} \gamma_t$ , where $\gamma_t$ now varies per token within a response, making the advantage token-specific even though the underlying reward is sequence-level. This is critical: the paper's contribution is precisely to introduce token-level variation into what would otherwise be a per-token constant advantage in GRPO.

Experimental configuration for RL experiments. The paper specifies the following hyperparameters for the RL training (Section 5.3.1, Appendix A.2):

Training batch size: 512 prompts per batch
Micro-batch size: 32, yielding 16 gradient accumulation steps per full batch
Learning rate: $1 \times 10^{-6}$
KL penalty: not used (the loss excludes both KL and entropy regularization)
Decoding temperature during training: $T = 1.0$
WAAD window: $W = 10$
FAI horizon: $H \in [10, 50]$ (lower bound 10, upper bound 50 tokens)
Context lengths: 1024 tokens for Countdown and QA; 1024 or 8192 for math reasoning
Training duration: 500 steps for 4B models, 600 steps for 8B models
Hardware: 8 GPUs for 4B training, 16 GPUs for 8B training
Models: Qwen3-4B-Base and Qwen3-8B-Base (both non-instruction-tuned base models)
Training data: DAPO-Math-17K for mathematical reasoning; 20,000 generated samples for Countdown; CrossThink-QA as provided

Why non-instruction-tuned base models: The paper deliberately uses base (non-SFT) models rather than instruction-tuned checkpoints because "using the non-SFT model avoids confounds from supervised instructions and provides a clean starting point for zero-RL training" (Section 4.1). This means the models have no prior exposure to chain-of-thought reasoning patterns — the RL training must elicit reasoning from scratch, making the effect of the credit assignment strategies more directly attributable.

Why the shorter 1024 context for some math experiments: The paper states that "the shorter context length is adopted as it yields a cleaner reasoning process that aligns more closely with the analytical environment and is less confounded by long-range dependencies that can dilute the effect of attention-based strategies" (Section 5.3.1). The 8192-context experiments demonstrate that the method scales to longer reasoning traces, but the 1024-context setting provides a controlled comparison where the attention signals are not complicated by very long-range dependencies.

Training data for mathematical reasoning. The paper uses DAPO-Math-17K (from Yu et al., 2025), a dataset of 17,000 mathematical problems with verifiable answers. This is notable because it is a relatively small dataset for RL training — the fact that the credit assignment strategies show consistent gains on this data volume suggests they improve data efficiency, though the paper does not make the data-efficiency claim explicit.

Evaluation protocol. The paper evaluates using the following metrics on each benchmark:

Countdown: accuracy on 512 held-out test instances
CrossThink-QA: accuracy on the provided test split
AIME24 and AIME25: average accuracy over 16 sampled responses per problem (avg@16)
AMC23, MATH500, OlympiadBench: pass@1 accuracy (single greedy or sampled response)

Plot smoothing. The training curves (Figures 4, 5, 6, 7) are smoothed using an Exponential Moving Average of the peak performance: $\text{EMA}_t = \alpha \cdot \max(x_t, \text{EMA}_{t-1}) + (1 - \alpha) \cdot \text{EMA}_{t-1}$ . This reflects a running average of the best performance seen so far, which emphasizes sustained improvements while filtering out transient fluctuations. The endpoint of each curve represents a relatively stable peak performance, and the method's advantage is visible both in earlier convergence (the curve rises faster) and higher final plateaus.

Design Choices and Their Justifications (Summary)

Head classification by quantile rather than fixed distance threshold: Adapts automatically to the model's attention span distribution without requiring manual threshold tuning; generalizes across different model sizes and architectures.
WAAD with clipped window $W=10$ : Isolates local structural patterns while preventing positional artifacts from dominating the signal; captures the typical phrasal chunk size in reasoning traces.
FAI with horizon window $[H_{\text{lo}}, H_{\text{hi}}]$ : Excludes immediate neighbors (which would inflate FAI for multi-token words) and caps the influence range to the region where anchoring effects are most pronounced.
Convergent validation via receiver head FAI: Independent head-filtering criteria (span-based vs. kurtosis-based) arriving at the same anchor tokens strengthens the claim that the anchor phenomenon is robust.
Perturbation experiments to establish causality: Moves beyond correlation (high FAI tokens are often attended to) to intervention (changing high FAI tokens alters reasoning trajectories), validating that FAI captures causally important positions.
Actor_attn as a separate instance with ego attention: Preserves flash attention efficiency in the inference engine while enabling attention map extraction with negligible overhead (one additional forward pass per response vs. thousands for generation).
Middle-layer sampling: Focuses attention extraction on layers most relevant to high-level reasoning structure, reducing computational and memory overhead.
Multiplicative credit scaling rather than additive bonus or separate loss: Integrates seamlessly into existing PPO/GRPO objectives without introducing new tradeoff hyperparameters.
Amplification applied only to nonnegative advantages: Prevents the credit assignment from amplifying penalties at structurally important positions that happened to produce poor outputs.
Coupled rhythm credit as back-allocation rather than joint amplification: Addresses the specific limitation of locally dominated anchors by shifting optimization pressure to the more flexible upstream preplan token, respecting the two-beat structure of the preplan-and-anchor rhythm.
Base (non-SFT) models as starting points: Removes confounds from supervised instruction data, providing a clean test of whether attention-derived credit assignment can guide reasoning emergence from scratch.
Moderate dataset sizes (DAPO-Math-17K, 20K Countdown): Tests the approach under realistic data constraints typical of RLVR workflows, where generating training data is expensive.

4. Key Insights and Innovations

Innovation 1: Attention Dynamics as a First-Class Diagnostic Substrate for Reasoning Structure

The paper's most fundamental intellectual move is elevating attention from a computational byproduct to a privileged diagnostic lens for understanding how LLMs organize reasoning. Prior work in mechanistic interpretability has studied attention extensively—identifying specialized head types (induction heads, iteration heads, receiver heads), mapping information flow through residual streams, and tracing deduction circuits. But these analyses have been predominantly descriptive and post-hoc: they explain what the model does after observing its behavior, but rarely produce signals that can be operationalized during training. The paper makes a decisive break from this pattern by treating attention not as an object of study but as a real-time sensor of the model's internal reasoning state—one that can be read during training and used to actively shape the optimization process.

What makes this move distinctive is the claim that attention reveals structure that is invisible to behavioral metrics. Prior approaches to identifying "important tokens" in reasoning have relied on external proxies: entropy (Wang et al., 2025b; Cheng et al., 2025) reflects the model's output uncertainty but conflates genuine decision points with superficial stylistic choices; content-based analysis (Bogdan et al., 2025) identifies semantically meaningful steps but requires interpreting the token's meaning, which is expensive and language-dependent; and contrastive estimation (Lin et al., 2024) compares trajectories to identify critical divergences but operates on behavioral outcomes rather than internal representations. The paper's key insight is that the model's own attention mechanism already encodes which tokens are structurally decisive—not as a trained behavior but as an emergent property of how transformer architectures route information. The WAAD and FAI metrics are not training a new classifier to identify important tokens; they are reading out a signal that the model computes anyway during inference.

This reframes the relationship between interpretability and optimization. The standard narrative treats interpretability as a diagnostic tool: first understand the model, then (maybe) apply insights to improve it. The paper collapses this loop by making interpretability directly actionable within the training process. The actor_attn instance (Section 5.1) is conceptually significant beyond its engineering role—it represents an architecture where the model's self-analysis (reading its own attention patterns) feeds back into its self-improvement (scaling advantages during RL). This is a step toward reflexive training systems where the model's internal organization guides its own optimization, rather than relying entirely on external reward signals.

The perturbation experiments (Table 1, Figure 3) are crucial for establishing that this diagnostic lens captures causal structure, not mere correlation. The finding that high-FAI token perturbations produce substantially lower Jaccard similarity (0.534 vs. 0.631, with 87.14% of trials showing greater deviation at high-FAI positions) demonstrates that FAI-identified anchors are causally necessary for the specific reasoning trajectory: changing them redirects downstream reasoning, while changing low-FAI tokens primarily affects surface phrasing. This moves beyond the typical interpretability claim ("the model pays attention to these tokens") to an interventionist claim ("these tokens govern the reasoning path"). Without this causal validation, the preplan-and-anchor rhythm would be merely an interesting descriptive pattern; with it, it becomes a mechanism that can be targeted for optimization.

The significance of this innovation extends beyond the specific WAAD/FAI metrics. It establishes a methodology template: (1) identify a structurally meaningful signal from attention dynamics, (2) validate its causal role through perturbation, (3) operationalize it as a training-time intervention within standard RL loops. This template can be applied to any attention-derived signal beyond preplan/anchor tokens, potentially opening a broad class of structure-aware optimization strategies. The paper's specific instantiation (WAAD/FAI → γt scaling) is the proof of concept; the conceptual framework is the lasting contribution.

Innovation 2: The Preplan-and-Anchor Rhythm as a Discovered Reasoning Primitive

The paper's second contribution is the empirical discovery and characterization of a specific, recurring reasoning rhythm—the preplan-and-anchor mechanism—that appears to organize how LLMs transition between reasoning chunks. This is not a hypothesis that the paper sets out to test; it is an emergent finding from the joint analysis of WAAD and FAI dynamics. The fact that these two independently computed metrics (one backward-looking, one forward-looking) exhibit systematic temporal coupling (FAI peaks following or coinciding with WAAD peaks at 52.53% vs. 36.87% random, a +42.47% lift; Table 2) suggests that the model has developed an internal cadence for reasoning that is not explicitly trained or prompted but arises from the interaction of local syntactic processing and global semantic organization.

The preplan-and-anchor rhythm is significant because it provides a unified framework for understanding phenomena that prior work observed in isolation. The local sawtooth pattern in attention maps—where attention reaches further back at chunk boundaries and collapses to the diagonal within chunks—has been noted in previous work on attention patterns (e.g., the observation that certain heads specialize in local vs. global processing), but was not connected to reasoning structure. The existence of "thought anchors"—tokens that persist in future attention and guide downstream inference—was documented by Bogdan et al. (2025) using receiver head analysis. The entropy peaks at reasoning junctures were noted by Wang et al. (2025b) and Cheng et al. (2025) as signals for exploration. But prior work treated these as separate observations with separate explanations. The preplan-and-anchor framework integrates them into a single mechanistic narrative: at a reasoning chunk boundary, the model recognizes local insufficiency (high entropy), consults long-range context (WAAD spike, preplan phase), and commits to a semantic anchor (FAI peak, anchor phase) that organizes subsequent processing.

This integration changes how we think about reasoning in LLMs. Rather than reasoning being a uniform process of "thinking step by step," the preplan-and-anchor rhythm suggests a two-beat structure where each reasoning step involves (1) deliberate, high-uncertainty context integration to determine the next direction, followed by (2) relatively automatic, low-uncertainty execution guided by a newly established anchor. This has implications beyond credit assignment: it suggests that interventions at the preplan phase (e.g., injecting additional context, modifying the retrieved information) might redirect reasoning more effectively than interventions at the anchor phase, and that the transition between the two phases represents a natural point for the model to "commit" to a particular reasoning path.

The locally dominated anchor concept (Section 4.2, Coupling Pattern 3) is a subtle but important conceptual contribution within this framework. It identifies a structural constraint on optimization: an anchor token that is strongly licensed by its immediately preceding preplan token has limited capacity for independent improvement because its generation is largely determined by the preplan context. This explains why simply amplifying credit at all high-FAI tokens (Strategy 2) is suboptimal—it wastes optimization signal on positions that cannot independently respond. The back-allocation mechanism in Strategy 3 directly addresses this constraint by recognizing that the functional unit of reasoning is not the individual token but the preplan-anchor pair, and that credit should be distributed across this pair according to optimization capacity. This is a fundamentally new way of thinking about credit assignment: rather than asking "which tokens were important?", it asks "which tokens in the important functional unit have the capacity to change, and how should credit flow to them?"

The quantitative validation of the three coupling patterns (Table 2) is methodologically significant because it establishes convergent validity: the same structural features emerge from multiple independent measurements (entropy, receiver-head FAI, global-head FAI, WAAD), ruling out the possibility that the preplan-and-anchor rhythm is an artifact of a particular metric or head selection criterion. The substantial lifts over random baselines (+42.47% to +171.49%) indicate genuine structure, but the fact that all lifts are well below 100% also indicates that the rhythm is a statistical tendency rather than a deterministic rule—there are many positions where WAAD peaks without high FAI, or high FAI without WAAD peaks. This nuance is important because it prevents over-interpretation: the preplan-and-anchor pattern is one organizing principle among many, not a complete description of reasoning structure.

Innovation 3: Structure-Aware Credit Assignment as a Bridge Between Interpretability and Optimization

The paper's third contribution is closing the loop between understanding and improving LLM reasoning through a practical, plug-and-play method that requires no architectural changes to existing RL pipelines. This is not an incremental improvement to RL algorithms—it is a category-defining contribution that opens a new axis of optimization: using model-internal signals to guide the distribution of learning pressure across tokens.

Prior work on improving RL for LLM reasoning has focused on two fronts: (1) better reward signals (process rewards, outcome rewards, learned verifiers), and (2) better optimization algorithms (PPO, GRPO, DPO, REINFORCE variants). These approaches treat the distribution of credit across tokens as a fixed consequence of the reward structure and algorithm design. Uniform credit assignment in GRPO, where the same group-normalized advantage is applied to every token in a response (Equation 6), is the extreme expression of this philosophy: the optimization signal is agnostic to internal reasoning structure because no structural information is available to the optimizer.

The paper fundamentally challenges this agnosticism by demonstrating that attention-derived signals can differentially guide credit assignment in a way that consistently improves performance across model scales, context lengths, and task types (Tables 3 and 4). The gains are not marginal—coupled rhythm credit achieves +10.5 points on Countdown over GRPO (from 52.6% to 63.1%) and +2.1 points averaged across mathematical reasoning benchmarks for Qwen3-4B-Base—and they are consistent (all three strategies outperform GRPO, with coupled credit consistently the strongest). This demonstrates that where you apply the optimization signal matters as much as the signal itself, and that attention dynamics provide a principled basis for deciding where to apply it.

The ablation results in Table 5 and Figure 6 provide critical validation of the structure-awareness claim. The finding that applying credit to the bottom 40% of tokens (by the same WAAD/FAI metrics) degrades performance relative to GRPO—with drops of -14.7 points on AMC23 for coupled bottom-40%—confirms that the metrics are not simply identifying "tokens where more credit helps regardless." They are identifying specific structural roles (preplan, anchor) where additional optimization pressure is beneficial, and distinguishing these from roles where it is harmful. The top-k ratio ablation (Figure 7, Table 5) further demonstrates that the identification of structurally decisive tokens is not binary—amplifying too few tokens (k=0.2) misses important signals, while amplifying too many (k≥0.6) dilutes the signal. The optimal k=0.4 reflects an empirical balance between coverage (including enough structurally important tokens) and concentration (not diluting the advantage amplification across too many positions).

The plug-and-play compatibility of the method (Section 5.1, Equation 12) is itself an innovation in research methodology. By designing the credit assignment as a multiplicative scaling of existing advantages (γt applied to At) rather than a new loss term or architectural modification, the paper makes the method immediately applicable to any PPO/GRPO-based RLVR pipeline. The minimal integration cost—adding an actor_attn instance and a γt computation step—means that researchers and practitioners can test and adopt the approach without rebuilding their training infrastructure. This design choice reflects a sophisticated understanding of how research advances in applied ML: methods that require minimal adoption friction have disproportionate impact because they can be tested, validated, and integrated at scale without prohibitive engineering cost.

The broader implication is that interpretability should be re-conceptualized as an input to optimization, not just an output of analysis. The paper demonstrates that interpretability insights can be translated into concrete training interventions that improve model performance, which fundamentally changes the value proposition of mechanistic interpretability research. Rather than interpretability being a separate research agenda focused on understanding models for understanding's sake, it becomes a tool for building better models—a feedback loop where understanding enables improvement, which enables further understanding. This closes the gap that the paper identifies in Section 2 between descriptive analyses of reasoning (which are "often descriptive and rarely yield process-aware training recipes") and prescriptive optimization strategies.

Innovation 4: Verifier-Free, Model-Intrinsic Credit Assignment as an Alternative to Learned Reward Models

A subtler but equally important innovation is that the paper's credit assignment mechanism is entirely model-intrinsic—it requires no external verifier beyond the standard sequence-level reward already used in RLVR, no process-level supervision, and no learned reward model that scores intermediate steps. The WAAD and FAI metrics are computed purely from the model's own attention maps during a single forward pass over the generated response. This is a fundamentally different approach to credit assignment than the dominant paradigm in reasoning RL, which has increasingly moved toward process reward models (PRMs) that score intermediate reasoning steps (Lightman et al., 2023; Wang et al., 2024; Shao et al., 2024).

PRMs represent the state of the art in credit assignment for reasoning, but they have significant limitations that this paper's approach circumvents. Training PRMs requires either expensive human annotations of step-level correctness (Lightman et al., 2023) or Monte Carlo rollouts from the base model (Wang et al., 2024), both of which are computationally intensive and introduce distribution shift as the policy improves. Moreover, PRMs can be gamed through reward hacking—the RL policy learns to produce steps that score well under the PRM without actually being correct—and require careful calibration to remain reliable under distributional shift. The paper's approach avoids these issues by not learning a separate verifier at all—the credit assignment signal comes from the model's own attention, which is always on-policy (since actor_attn is synchronized with actor_train) and cannot be "hacked" in the same way because it is not an optimization target (γt is detached from gradients).

This positions the paper's method as an alternative paradigm for fine-grained credit assignment that is orthogonal to and potentially complementary with PRM-based approaches. PRMs assign credit based on what the model wrote (is this step correct?); WAAD/FAI assign credit based on how the model organized its internal processing while writing it (was this token a preplan or anchor?). These are different sources of information that could be combined—for example, using PRM scores to determine whether a reasoning step is correct, and using WAAD/FAI to determine how to distribute credit within that step across preplan and anchor tokens. The paper does not explore this combination, but the conceptual framework establishes it as a natural direction.

The self-supervised nature of the credit assignment also means the method generalizes across tasks without task-specific verifier design. The paper demonstrates consistent gains across Countdown puzzles (rule-based verification), CrossThink-QA (open-ended QA), and mathematical reasoning (execution-based verification), with the same WAAD/FAI metrics and credit assignment strategies applied uniformly. This suggests that the preplan-and-anchor rhythm may be a domain-general feature of LLM reasoning rather than a math-specific phenomenon, though the paper does not test on non-reasoning tasks. If this generality holds, it would mean that structure-aware credit assignment could be applied to any RLVR task without designing task-specific process supervision, which would substantially lower the barrier to adoption compared to PRM-based approaches.

The perturbation experiments (Section 4.1, Table 1) are particularly important for this innovation because they establish that FAI captures causal influence on reasoning outcomes without requiring any outcome signal. The Jaccard similarity comparison shows that high-FAI tokens shape downstream reasoning content, but this is measured purely through trajectory divergence—no correctness labels are needed to identify which tokens matter. This means the credit assignment signal is available even on tasks where correctness is ambiguous or multi-dimensional, as long as the model exhibits structured attention patterns. The paper does not test this on open-ended generation tasks, but the conceptual framework suggests applicability beyond verifiable-reward settings.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three categories of reasoning tasks: (i) the Countdown puzzle (Pan et al., 2025), a symbolic reasoning task where the model must combine four given integers using arithmetic operations to reach a target value, with a test set of 512 instances and a training set of 20,000 synthetically generated samples; (ii) CrossThink-QA (Akter et al., 2025), a multi-domain QA dataset aggregating questions from heterogeneous sources including science, commonsense, and trivia; and (iii) five mathematical reasoning benchmarks: AIME24 and AIME25 (competition problems from the 2024 and 2025 American Invitational Mathematics Examination), AMC (Li et al., 2024, problems from the American Mathematics Competitions), MATH500 (Hendrycks et al., 2021, a 500-problem subset of the MATH dataset spanning algebra, geometry, number theory, and combinatorics), and OlympiadBench (He et al., 2024, international mathematical olympiad-level problems). For mathematical reasoning RL training, the paper uses DAPO-Math-17K (Yu et al., 2025), a dataset of 17,000 problems with verifiable answers. This diverse benchmark suite tests both structured symbolic reasoning and open-ended mathematical problem-solving across a wide difficulty range, from relatively accessible puzzles to competition-level mathematics.
Base model(s). Experiments use Qwen3-4B-Base and Qwen3-8B-Base (Yang et al., 2025a), both non-instruction-tuned base models. The paper deliberately selects base models rather than SFT checkpoints because "using the non-SFT model avoids confounds from supervised instructions and provides a clean starting point for zero-RL training" (Section 4.1), ensuring that any reasoning behaviors observed are emergent from the RL process rather than inherited from supervised fine-tuning. The two model scales (4B and 8B parameters) test whether the attention-derived signals generalize across model sizes. Qwen3-4B-Base is used for the detailed attention dynamics analysis in Section 4 (with GSM8K prompts and temperature T=0.7), while both models are used for the RL experiments in Section 5.
Metrics. Evaluation metrics vary by benchmark: for Countdown, accuracy on 512 held-out test instances; for CrossThink-QA, accuracy on the provided test split; for AIME24 and AIME25, average accuracy over 16 sampled responses per problem (avg@16); for AMC23, MATH500, and OlympiadBench, pass@1 accuracy (single sampled or greedy response). Training curves (Figures 4–7) are smoothed using an Exponential Moving Average of peak performance: EMAt = α · max(xt, EMAt−1) + (1 − α) · EMAt−1, which produces a running average of the best performance observed so far. The smoothed curves emphasize sustained improvements while filtering transient fluctuations; the endpoint of each curve represents a relatively stable peak.
Baselines. The primary baseline is GRPO (Shao et al., 2024), the standard on-policy RL algorithm that estimates advantages through group normalization (Equation 6: the advantage for each response is its reward minus the group mean, divided by the group standard deviation, applied uniformly across all tokens). This is the direct target for comparison since the paper's method replaces uniform advantages with shaped advantages. Two additional baselines isolate the effect of token selection from the effect of attention-derived signals: Random credit randomly selects tokens for advantage amplification (same amplification factor γamp=1.5, same top-40% quantile), testing whether any non-uniform credit distribution helps regardless of which tokens are selected; and High-entropy credit amplifies advantages at tokens with high predictive entropy Ht = −∑v pθ(v | q, o<t) log pθ(v | ·), following the line of work (Wang et al., 2025b; Cheng et al., 2025; Cui et al., 2025) that identifies high-entropy tokens as important for reasoning exploration. These baselines are critical for establishing that the WAAD/FAI-based selection provides value beyond simply breaking uniform credit assignment or using an alternative behavioral signal.
Generation budget / compute accounting. The paper does not directly measure FLOPs or wall-clock time; instead, compute is implicitly measured in terms of training steps (500 steps for 4B models, 600 steps for 8B models) with fixed batch sizes. All methods—GRPO and the three credit-shaping variants—use identical training infrastructure, batch sizes (512 prompts per batch, micro-batch size 32 yielding 16 gradient steps per batch), and number of training steps, making the comparison fair in terms of total optimization steps. The additional cost of the credit assignment method comes from the actor_attn forward pass (one additional forward pass per generated response to extract attention maps), which the paper argues is negligible compared to the thousands of forward passes required for autoregressive generation. The paper does not report absolute wall-clock time or GPU-hours for experiments.
Cross-validation / statistical protocol. The paper does not employ cross-validation or multiple random seeds in the RL experiments. The training curves (Figures 4–7) show smoothed performance over training steps for single runs. The perturbation experiments in Section 4.1 (Table 1) use 70 randomly sampled math problems and compare top-k vs. bottom-k perturbations; statistical significance is reported as the percentage of trials where top-k Jaccard similarity is lower than bottom-k (87.14%). For the coupling pattern analysis (Table 2), significance is assessed by comparing observed coupling statistics against random baselines computed by shuffling peak positions while preserving their count, with lift calculated as (Observed − Random) / Random. No confidence intervals or standard deviations are reported for the main RL benchmark results (Tables 3 and 4), which is a limitation for assessing the reliability of the reported gains.

Main Quantitative Results

Results on Simple Logical Puzzles and QA (Table 3, Figure 4)

Table 3 reports final accuracy on the Countdown puzzle and CrossThink-QA benchmarks. On Countdown, GRPO achieves 52.6%. The random credit baseline provides a marginal improvement to 55.0%, and high-entropy credit reaches 57.7%. The three attention-derived strategies show substantially stronger gains: local-chunk credit achieves 59.9% (+7.3 points over GRPO), global-anchor credit achieves 60.4% (+7.7 points), and coupled rhythm credit achieves the strongest result at 63.1% (+10.5 points). This progression—random < entropy < local-anchor < global-anchor < coupled—is consistent with the paper's claim that attention-derived signals capture structurally meaningful token importance beyond what generic behavioral signals (entropy) or random selection can provide, and that the coupled rhythm strategy that integrates both preplan and anchor signals with back-allocation outperforms either signal individually.

On CrossThink-QA, where reasoning is more open-ended and less constrained by formal rules, the absolute gains are smaller but the pattern persists. GRPO achieves 48.0%. Random credit slightly decreases performance to 47.8% (within noise), while high-entropy credit matches GRPO at 48.0%. Local-chunk credit reaches 50.0% (+2.0), global-anchor credit reaches 49.6% (+1.6), and coupled rhythm credit reaches 50.1% (+2.1). The compression of gains (max +2.1 on QA vs. +10.5 on Countdown) likely reflects the more heterogeneous nature of QA reasoning, where the preplan-and-anchor rhythm may be less consistently present across diverse question types, but all credit-aware variants still show consistent improvement over GRPO.

Figure 4 shows the training curves for both tasks, with Qwen3-4B-Base on Countdown and Qwen3-8B-Base on QA. The curve dynamics reveal two patterns: faster convergence—the credit-shaped variants rise earlier in training than GRPO, with coupled rhythm credit showing the earliest separation from the baseline—and higher final plateaus—GRPO curves flatten at lower levels while the credit-shaped variants continue to improve or stabilize at higher levels. The smoothed EMA curves show that coupled rhythm credit achieves both the highest peak and the most sustained improvement across training steps.

Results on Mathematical Reasoning (Table 4, Figure 5)

Table 4 presents the core mathematical reasoning results across three experimental configurations: Qwen3-4B-Base with 1K context length, Qwen3-8B-Base with 1K context length, and Qwen3-4B-Base with 8K context length. Within each configuration, results are reported for five benchmarks and an average across benchmarks.

Qwen3-4B-Base with 1K context. GRPO achieves an average of 37.1% across the five benchmarks. Random credit matches this at 37.1% and high-entropy credit shows minimal improvement to 37.2%, confirming that generic token selection provides no benefit at this scale and task distribution. Global-anchor credit raises the average to 38.2% (+1.1), with gains distributed across all benchmarks: +0.9 on AIME24, +0.6 on AIME25, +2.5 on AMC23, +1.2 on MATH500, and +0.2 on OlympiadBench. Local-chunk credit achieves an average of 38.6% (+1.5), with a particularly strong +2.1 on AIME24 and +3.3 on AMC23 but a smaller +0.7 on MATH500, suggesting local-chunk credit may be particularly effective on problems requiring structured multi-step planning (AIME, AMC) versus those where within-chunk computation dominates (MATH500). Coupled rhythm credit achieves the highest average at 39.2% (+2.1), with the largest improvements on AIME25 (+2.6) and AIME24 (+2.3), and consistent gains across all benchmarks.

Qwen3-8B-Base with 1K context. The larger model substantially raises all baseline scores: GRPO averages 39.4%. Random credit again provides minimal benefit (39.6%), and high-entropy credit modestly improves to 40.1% (+0.7 over GRPO). Global-anchor credit achieves 42.1% (+2.7), local-chunk credit reaches 41.3% (+1.9), and coupled rhythm credit reaches the strongest result at 43.2% (+3.8). The gains on individual benchmarks are notably larger in absolute terms than at 4B scale: coupled rhythm credit delivers +5.0 on AIME25, +6.3 on AMC23, and +2.8 on OlympiadBench. This scaling of absolute gains with model size is important—it suggests that the preplan-and-anchor rhythm becomes more pronounced or more optimizable as the base model's reasoning capabilities improve, rather than being a small-model artifact.

Qwen3-4B-Base with 8K context. This configuration tests whether the credit assignment strategies remain effective when the model generates longer reasoning traces (the paper notes that the 8K context setting "already benefits from longer reasoning traces"). GRPO with 8K context achieves 44.8% average, substantially higher than the 37.1% with 1K context, confirming that longer reasoning improves baseline performance. Random credit (44.7%) and high-entropy credit (44.5%) are slightly below GRPO, indicating that non-selective credit perturbation may be slightly harmful when the baseline is already strong. Global-anchor credit achieves 47.1% (+2.3), local-chunk credit reaches 47.2% (+2.4), and coupled rhythm credit reaches 47.3% (+2.5). The gains are compressed relative to the 1K context results (max +2.5 vs. +3.8 at 8B-1K), which may reflect ceiling effects or the fact that longer reasoning traces dilute the influence of individual preplan/anchor tokens since there are more tokens total. Nevertheless, all three attention-derived strategies consistently outperform GRPO, demonstrating robustness to sequence length.

Figure 5 shows the training curves for all three experimental configurations. The curves consistently show credit-shaped variants separating from GRPO early in training and maintaining higher plateaus. The coupled rhythm credit curves are consistently the top performers across all configurations, and the gap between coupled credit and GRPO is most pronounced on the larger model (Qwen3-8B), where the absolute improvement is largest.

Ablation: Top-k vs. Bottom-k Credit Assignment (Figure 6, Table 5)

This set of experiments tests a critical question: are the WAAD and FAI metrics genuinely identifying tokens where additional credit is beneficial, or would any non-uniform credit distribution help? Figure 6 and Table 5 compare top-40% credit assignment (amplifying advantages at the highest-scoring tokens by WAAD/FAI) against bottom-40% credit assignment (amplifying advantages at the lowest-scoring tokens) on Qwen3-8B-Base with 1K context.

The results are stark. GRPO baseline achieves 39.4% average across the five math benchmarks. Global-anchor credit, bottom-40%: 33.1% average, a −6.3 point drop from GRPO, with devastating drops on AMC23 (−13.1) and OlympiadBench (−7.8). Local-chunk credit, bottom-40%: 37.6% average, a −1.8 point drop, with more moderate but consistent degradation across benchmarks. Coupled rhythm credit, bottom-40%: 32.5% average, a −6.9 point drop, with AMC23 dropping by −14.7 points. In contrast, the top-40% variants all improve over GRPO (global: +2.7, local: +1.9, coupled: +3.7).

This asymmetry is fundamental to the paper's claims. It demonstrates that the WAAD/FAI metrics are not simply identifying "tokens where additional optimization pressure has any effect" but specifically identifying tokens where additional credit is beneficial. Reinforcing credit on tokens with low WAAD drop (within-chunk continuations) or low FAI (tokens with minimal downstream influence) actively degrades performance, presumably because it encourages the model to over-optimize routine, structurally unimportant positions at the expense of the planning and anchoring behaviors that actually drive correct reasoning. This provides strong evidence that the preplan-and-anchor structure is not just a descriptive pattern but a normative guide for where optimization should focus.

Figure 6 shows the peak performance moving averages for top-k vs. bottom-k credit assignment. The top-k curves consistently rise above the GRPO baseline, while the bottom-k curves either stagnate at GRPO levels (local-chunk bottom) or fall substantially below (global and coupled bottom). This visual separation confirms that the direction of credit assignment (toward structurally important tokens vs. away from them) matters critically.

Ablation: Top-k Ratios (Figure 7, Table 5)

This ablation varies the quantile threshold q (the fraction of tokens selected for advantage amplification) for the coupled rhythm credit strategy on Qwen3-8B-Base with 1K context. Five ratios are tested: q = 0.2 (top 20%), q = 0.4 (top 40%, the default), q = 0.6 (top 60%), q = 0.8 (top 80%), and q = 1.0 (all tokens, equivalent to uniform amplification of all advantages by 1.5×).

Table 5 reports the per-benchmark and average results. Coupled rhythm credit with q = 0.4 achieves the highest average at 43.2%. Reducing to q = 0.2 decreases the average to 40.6% (a drop of −2.6 from q = 0.4), with particularly large drops on AMC23 (−6.7 from 65.4 to 58.7). Increasing to q = 0.6 decreases the average to 40.2% (−3.0); q = 0.8 yields 40.7% (−2.5); and q = 1.0 (uniform amplification) yields 39.8%, which is only marginally above the GRPO baseline of 39.4%. Figure 7 shows the performance curves for each ratio.

These results reveal a non-monotonic relationship between credit concentration and performance. Amplifying too few tokens (q = 0.2) likely misses structurally important preplan/anchor tokens that fall below the top-20% threshold, leaving them without the optimization boost they need. Amplifying too many tokens (q ≥ 0.6) dilutes the advantage amplification across positions that are structurally unimportant, reducing the effective gradient signal at the truly critical positions. The fact that q = 1.0 (uniform 1.5× amplification of all tokens) performs near GRPO levels confirms that the selection of specific tokens matters more than the magnitude of amplification—simply multiplying all advantages by 1.5 provides negligible benefit because it does not change the relative distribution of optimization pressure across tokens.

The optimal q = 0.4 reflects an empirical balance: the top 40% of tokens by WAAD/FAI metrics capture the structurally decisive positions (preplan tokens at chunk boundaries, anchor tokens with high downstream influence) while excluding the majority of routine continuation tokens where additional credit is wasteful or harmful. The sharp performance drop from q = 0.4 to q = 0.2 suggests that structurally important tokens are relatively broadly distributed across the reasoning trace rather than concentrated in a small fraction of positions, consistent with the observation that reasoning involves multiple preplan-anchor cycles distributed throughout the trajectory.

Ablation Studies and Robustness Checks

Random credit baseline (Tables 3, 4): Across all settings, random token selection for credit amplification yields marginal or no improvement over GRPO. On Qwen3-4B-Base with 1K context (Table 4), random credit matches GRPO exactly at 37.1% average; on Qwen3-8B-Base with 1K context, random credit achieves 39.6% vs. GRPO's 39.4% (+0.2); on Qwen3-4B-Base with 8K context, random credit slightly underperforms at 44.7% vs. 44.8% (−0.1). This establishes that the gains from the attention-derived strategies are not attributable to simply breaking uniform credit assignment—the specific choice of which tokens to amplify is essential.
High-entropy credit baseline (Tables 3, 4): The entropy-based strategy, representing the state of the art in behavioral token importance identification, provides small improvements over GRPO in some settings but consistently underperforms the attention-derived strategies. On Countdown (Table 3), high-entropy credit achieves 57.7% (+5.1 over GRPO) compared to 60.4% for global-anchor (+7.7) and 63.1% for coupled (+10.5). On mathematical reasoning (Table 4), high-entropy gains are marginal: +0.1 on 4B-1K (37.2% vs. 37.1%), +0.7 on 8B-1K (40.1% vs. 39.4%), and −0.3 on 4B-8K (44.5% vs. 44.8%). This demonstrates that WAAD/FAI capture structural information beyond what entropy alone provides—entropy conflates genuine preplan tokens (where the model is uncertain because it must decide the next reasoning direction) with superficial high-entropy tokens (where the model is uncertain for arbitrary reasons), while WAAD specifically identifies the context-consultation behavior that characterizes preplan tokens.
Model scale robustness (Table 4): The credit assignment strategies show consistent gains across both 4B and 8B model scales, with larger absolute improvements at the larger scale. For coupled rhythm credit: +2.1 average on 4B-1K, +3.8 on 8B-1K, and +2.5 on 4B-8K. The larger gains at 8B suggest that as the base model's reasoning capabilities improve, the preplan-and-anchor rhythm becomes more pronounced or more amenable to optimization, though the paper does not provide mechanistic evidence for why this scaling occurs.
Context length robustness (Table 4): Comparing Qwen3-4B-Base at 1K vs. 8K context lengths, coupled rhythm credit achieves +2.1 average at 1K and +2.5 at 8K. While the 8K baseline is substantially higher (44.8% vs. 37.1%), the relative improvement is comparable, indicating that the credit assignment strategy remains effective even when reasoning traces are longer and the attention signals are diluted across more tokens. The slight reduction in relative gain magnitude (from roughly 5.7% relative improvement at 1K to 5.6% at 8K) is small.
Task type robustness (Tables 3, 4): The method demonstrates gains across structured symbolic reasoning (Countdown, +10.5), open-ended QA (CrossThink-QA, +2.1), and mathematical reasoning (average +2.1 to +3.8 depending on configuration). The compressed gains on QA likely reflect the more heterogeneous nature of QA reasoning, where the preplan-and-anchor rhythm may be less consistently present. The paper does not analyze attention patterns on QA to determine whether the rhythm persists in that domain, which is a limitation.
Coupled credit outperforms individual strategies (Tables 3, 4): Across all settings, coupled rhythm credit achieves the highest performance, with global-anchor typically second and local-chunk third. On 8B-1K (Table 4), coupled credit (43.2%) > global (42.1%) > local (41.3%) > GRPO (39.4%). On 4B-8K, coupled (47.3%) ≈ local (47.2%) > global (47.1%) > GRPO (44.8%). The consistent superiority of coupled credit validates the joint consideration of preplan and anchor signals and the back-allocation mechanism: integrating both signals provides information beyond either individually, and reallocating credit from locally dominated anchors to their preplan tokens addresses the limited optimization capacity of locally licensed anchor positions.
Locally dominated anchor mechanism (qualitative, Section 5.2): The paper's coupled rhythm credit strategy relies on the concept of locally dominated anchors—high-FAI tokens whose own generation is strongly determined by a preceding WAAD peak, making them poor targets for independent optimization. The paper does not provide a direct ablation showing that the back-allocation component specifically (i.e., the α parameter controlling how much credit shifts from anchor to preplan) improves over a version of coupled credit without back-allocation. This is a notable missing ablation: the coupled credit strategy combines three elements (preplan amplification, anchor amplification, and back-allocation), and the paper does not isolate the contribution of back-allocation separately.
γamp sensitivity (no explicit ablation): The paper uses a fixed amplification factor γamp = 1.5 across all experiments and strategies. No ablation is provided for different γamp values (e.g., 1.2, 2.0, 3.0). The paper's top-k ratio ablation (Figure 7, Table 5) varies the selection threshold but keeps γamp fixed, leaving open the question of whether a different amplification factor would shift the optimal k or improve performance. The choice of 1.5 appears to be a reasonable default, but the robustness of results to this hyperparameter is not established.
Training duration (500–600 steps): The paper trains 4B models for 500 steps and 8B models for 600 steps. The training curves (Figures 4, 5, 7) show that the GRPO baseline plateaus before this point in most cases, and the credit-shaped variants stabilize at higher levels, suggesting that the reported results represent converged performance rather than transient advantages during training. However, no experiments extend training significantly beyond these step counts to verify that the gains persist or to test whether different methods would eventually converge to similar performance with enough training.
Dataset size (DAPO-Math-17K): The mathematical reasoning experiments use a relatively small training dataset of 17,000 problems. The paper does not provide experiments varying dataset size to test whether the credit assignment strategies provide larger gains under data-scarce conditions (where efficient use of each training sample matters more) or whether the gains persist with substantially larger datasets. This is a missed opportunity to characterize the data-efficiency benefits that the paper claims motivationally in Section 1.

Critical Assessment

The experiments provide credible evidence that attention-derived credit assignment can improve RL training efficiency for LLM reasoning, but several aspects of the paper's central claims require qualification based on what was and was not tested.

Claim: "Attention dynamics reveal an intrinsic preplan-and-anchor rhythm in LLM reasoning." The empirical evidence for this claim rests on three quantitative coupling patterns (Table 2) and the perturbation experiments (Table 1). The coupling patterns show statistically significant lifts over random baselines, establishing that WAAD peaks, FAI peaks, and entropy peaks co-occur more often than chance would predict. However, the lifts are substantial but not dominant: WAAD-FAI alignment is 52.53% vs. 36.87% random (+42.47% lift), meaning that nearly half of all reasoning positions do not exhibit the coupled rhythm. The paper characterizes the rhythm as "recurring" rather than universal, which is appropriate, but the strength of the claim should be tempered by the observation that the coupling is a statistical tendency rather than a deterministic mechanism operating at every reasoning boundary.

The perturbation experiments (Table 1) establish that high-FAI tokens causally influence downstream reasoning content, but this demonstrates that anchors matter, not specifically that the preplan-and-anchor two-beat rhythm is the operative mechanism. The perturbation analysis targets isolated high-FAI positions; it does not test whether perturbing a preplan token (WAAD peak) has different effects than perturbing the subsequent anchor (FAI peak), or whether perturbing both in combination produces effects that differ from perturbing either alone. Such experiments would provide direct evidence for the preplan-anchor coupling as a functional unit rather than as independent important positions that happen to co-occur.

The analysis is conducted exclusively on Qwen3-4B-Base with GSM8K prompts. While GSM8K is a standard math reasoning benchmark, it is a relatively narrow distribution of grade-school word problems. The paper does not demonstrate the preplan-and-anchor rhythm on the other benchmarks (Countdown, CrossThink-QA, AIME, AMC, MATH500, OlympiadBench) that are used for the RL experiments. If the rhythm is specific to GSM8K-style reasoning or to the 4B model scale, the RL gains on other benchmarks might be attributable to a different mechanism than the one described. The paper's claim that the rhythm is intrinsic to LLM reasoning would be stronger with attention analyses on at least one additional benchmark or model scale.

Claim: "Structure-aware RL strategies that amplify credit at preplan and anchor tokens consistently improve over uniform credit assignment." The experimental results in Tables 3 and 4 robustly support this claim across two model scales (4B and 8B), two context lengths (1K and 8K), and three task categories (puzzles, QA, mathematical reasoning). The gains are consistent: every attention-derived strategy outperforms GRPO in every configuration, and the coupled rhythm strategy is always the best or tied for best. The bottom-k ablation (Table 5) provides strong evidence that the WAAD/FAI metrics specifically identify beneficial credit amplification targets, since amplifying low-scoring tokens consistently degrades performance.

However, the absolute magnitude of gains varies substantially across tasks and configurations. On Countdown, the gains are large (+10.5 points over GRPO at 52.6%, a 20% relative improvement). On mathematical reasoning, the gains are more modest but consistent: +2.1 average on 4B-1K (5.7% relative), +3.8 on 8B-1K (9.6% relative), and +2.5 on 4B-8K (5.6% relative). On CrossThink-QA, gains are minimal (+2.1 points, 4.4% relative). This variation raises questions about what factors modulate the effectiveness of the method: task structure (formal rules vs. open-ended reasoning), base model capability, or the presence/strength of the preplan-and-anchor rhythm in the model's attention patterns on those tasks. The paper does not provide analysis linking the magnitude of RL gains to the strength of the preplan-and-anchor rhythm on different tasks, which would be informative for predicting when the method will be most beneficial.

Claim: "The method is plug-and-play with standard RLVR." This claim is supported by the experimental design: the paper implements the credit assignment strategies on top of the ROLL framework (Wang et al., 2025c) using GRPO, and the only modification is replacing uniform advantages with shaped advantages (Atγt). No architectural changes, new loss terms, or modifications to the RL algorithm are required. The three-model architecture (actor_infer, actor_attn, actor_train) introduces implementation complexity but does not change the training objective or optimization procedure.

However, the "plug-and-play" characterization requires qualification. The method requires an additional model instance (actor_attn) with eager attention to extract attention maps, and the infrastructure to synchronize weights across three instances. For research groups or practitioners not already using a multi-instance training setup (which is common in large-scale RL training with separate inference and training engines), this represents a non-trivial integration cost. The paper's claim that overhead is "minimal" rests on the comparison of one additional forward pass to thousands of generation forward passes, which is correct for per-response cost but does not account for the memory overhead of maintaining a separate model instance with full attention matrices. No wall-clock time measurements or memory usage comparisons are provided.

Missing experiments that would strengthen the paper:

Multiple random seeds for RL experiments. The paper reports single-run results without confidence intervals or standard deviations. With benchmark datasets of modest size (512 Countdown test instances, 500 MATH500 problems), the reported gains may be sensitive to random seed variation. Multi-seed experiments would establish the statistical reliability of the improvements.
Comparison to process reward model (PRM) baselines. The paper argues (Section 2) that PRMs are expensive and susceptible to reward hacking, motivating the attention-derived approach as a verifier-free alternative. However, no direct comparison is provided against PRM-based credit assignment, which is the dominant paradigm for fine-grained credit in reasoning RL. Such a comparison would contextualize the gains: are these strategies competitive with or complementary to PRM-based approaches?
Ablation of the back-allocation component in coupled rhythm credit. The coupled rhythm strategy combines preplan amplification, anchor amplification, and back-allocation from locally dominated anchors. An ablation comparing coupled credit with and without back-allocation (α = 0 vs. α > 0) would isolate the contribution of the locally dominated anchor mechanism, which is the most conceptually novel aspect of Strategy 3.
Attention analysis on non-GSM8K tasks. The paper's mechanistic claims about the preplan-and-anchor rhythm are based on GSM8K analysis, but the RL experiments span multiple benchmarks. Demonstrating that the rhythm appears on at least one additional benchmark (e.g., MATH or Countdown) would strengthen the link between the described mechanism and the observed RL gains.
Scaling to larger models. The experiments use 4B and 8B parameter models. Testing on larger scales (e.g., 14B, 32B, or 70B) would establish whether the preplan-and-anchor rhythm and the corresponding credit assignment benefits persist or change as model capability increases. The paper shows that gains are larger in absolute terms at 8B compared to 4B, which is suggestive but limited to a single doubling of scale.
Data efficiency experiments. The paper's motivation (Section 1) claims that structure-aware credit assignment can improve data efficiency, but no experiments vary the training dataset size to test whether the credit-shaped strategies achieve equivalent performance with less data, or greater performance with the same data. The fixed 17K training set (DAPO-Math-17K) and 500–600 training steps do not address this question.
γamp sensitivity analysis. The amplification factor is fixed at 1.5 across all experiments without ablation. Testing values from 1.1 to 3.0 would establish whether the method is sensitive to this hyperparameter and whether the optimal factor interacts with model scale, task type, or credit assignment strategy.
Analysis of when the method fails. The paper does not report cases where the credit assignment strategies underperform GRPO, beyond the bottom-k ablation. Real-world deployment of this method would benefit from understanding whether there are prompt types, reasoning patterns, or training regimes where attention-derived credit assignment is neutral or harmful.

Summary of evidential strength and weaknesses. The paper's experimental case for the effectiveness of attention-derived credit assignment is internally consistent and shows gains across a well-chosen range of configurations. The bottom-k and top-k ratio ablations provide strong evidence that the WAAD/FAI metrics specifically identify beneficial amplification targets. However, the single-run results without confidence intervals, the absence of comparison to PRM baselines, the lack of attention analysis on non-GSM8K tasks, and the missing ablation on the back-allocation mechanism leave important questions about statistical reliability, relative effectiveness, mechanistic grounding, and component contributions unanswered. The paper convincingly demonstrates that attention-derived credit assignment can improve RL training for LLM reasoning; it does not establish how much of that improvement comes specifically from the preplan-and-anchor mechanism versus from generic concentration of credit on high-influence tokens, how the method compares to alternative fine-grained credit assignment approaches, or how broadly the underlying attention rhythm generalizes across tasks and scales.

6. Limitations and Trade-offs

6.1 The Preplan-and-Anchor Rhythm Is Demonstrated on a Single Model and Single Task Family

The assumption or constraint. The paper's entire mechanistic analysis—head classification, WAAD/FAI metric validation, coupling pattern quantification, and perturbation experiments—is conducted exclusively on Qwen3-4B-Base using prompts from the GSM8K dataset (Section 4.1, Appendix A.1). The paper explicitly acknowledges this scope: "We analyze attention dynamics using Qwen3-4B-Base on math prompts from the GSM8K dataset" (Section 4.1), but does not extend the attention analysis to any other model scale (the 8B model is used only in RL experiments, not in attention dynamics analysis), any other task family (Countdown, CrossThink-QA, competition math), or any other model architecture.

The consequence. The paper's central mechanistic claim—that LLMs exhibit an intrinsic preplan-and-anchor rhythm visible through attention dynamics—rests on evidence from a single model-task combination. Several failure modes are possible:

The rhythm may be Qwen3-specific. The paper notes that "using the non-SFT model avoids confounds from supervised instructions" (Section 4.1), but this means the analysis is conducted on a model that has never been trained to follow instructions or produce structured chain-of-thought. The preplan-and-anchor pattern might be an artifact of the base model's pretraining distribution rather than a general feature of how LLMs organize reasoning. Models with different pretraining data mixtures, tokenizers, or architectural details (e.g., different numbers of attention heads, different layer counts) might exhibit different attention dynamics that affect whether WAAD and FAI capture the same preplan-anchor structure.
The rhythm may be GSM8K-specific. GSM8K consists of grade-school math word problems that follow highly stereotyped templates (a context paragraph, a question, a step-by-step arithmetic solution). The preplan-and-anchor rhythm might reflect the model's adaptation to this specific problem structure rather than a domain-general reasoning mechanism. The paper observes (Section 6.1) that the RL gains are largest on Countdown (+10.5 points) and smaller on CrossThink-QA (+2.1 points), which could indicate that the attention-derived signals are less informative or the credit assignment strategies less effective when the reasoning structure is more heterogeneous—but without attention analysis on those tasks, this remains speculation.
The rhythm may not scale to larger models. The 4B analysis shows the preplan-and-anchor pattern, and the RL experiments at 8B show that the credit assignment strategies work at that scale, but the paper does not demonstrate that the same WAAD/FAI dynamics are present in the 8B model's attention maps or that the mechanism persists at larger scales where reasoning capabilities are qualitatively different. Larger models may develop more sophisticated attention patterns that render the simple local/global dichotomy and the preplan-anchor rhythm less descriptive of actual reasoning organization.

What evidence exists in the paper. The RL results in Tables 3 and 4 show that the credit assignment strategies produce gains across multiple benchmarks and both model scales, but this demonstrates that the credit assignment works, not that the underlying mechanism described in Section 4 is what drives the gains. The gains could arise from a different reason—for example, WAAD and FAI might simply identify tokens that the model considers important for any reason, and amplifying credit at those positions is beneficial regardless of whether they form a preplan-anchor rhythm. The bottom-k ablation (Table 5) confirms that high-WAAD-drop and high-FAI tokens are better amplification targets than low-scoring tokens, but does not confirm that the temporal coupling (WAAD peaks preceding FAI peaks) or the two-beat structure specifically are necessary for the gains. A practitioner cannot be confident that the method will generalize to their model or task without replicating the attention analysis.

Mitigation status. The paper does not address this limitation. The attention analysis is presented as a fixed, one-time validation of the metrics and patterns, and the RL experiments test the downstream utility of the metrics without verifying that the same patterns underlie the gains on the RL benchmarks. Section 8 (Conclusion) states broadly that "attention both explains intrinsic model behaviors and prescribes targeted interventions," but does not acknowledge the gap between the single-task mechanistic evidence and the multi-task optimization results. Future work could replicate the attention analysis on additional benchmarks and model scales to establish generality, or could provide correlational evidence linking the strength of the preplan-and-anchor rhythm on a given task to the magnitude of RL gains.

6.2 The Difficulty Estimation Cost Is Unaccounted for in Headline Efficiency Claims (Difficulty Estimation Overhead for Attention Extraction)

The assumption or constraint. The credit assignment strategies require extracting full attention maps from the model during training, which the paper accomplishes through a dedicated actor_attn instance running a standard (non-flash) Transformer forward pass (Section 5.1). The paper characterizes this overhead as minimal: "The model typically requires thousands of forward passes for token-by-token generation, whereas we obtain attention maps with just a single additional forward pass over the generated response, introducing little additional latency with parallel computation."

The consequence. This characterization is incomplete in several ways that matter for practical deployment:

Memory overhead for eager attention. Flash attention's primary advantage is reducing memory complexity from O(N²) to O(N) for sequence length N. The actor_attn instance, by using eager attention, must materialize the full N×N attention matrix (or at minimum the attention patterns for the sampled layers). For the 8K context length experiments, this means attention maps of size 8192×8192 per sampled layer—approximately 268 million floating-point values per map for 32-bit precision, or roughly 1 GB per attention map. With five sampled layers and batch processing, this becomes a substantial memory burden that the paper does not quantify. The comparison to "thousands of forward passes" for generation is misleading because the generation forward passes use memory-efficient flash attention, while the single analysis pass does not.
No wall-clock or memory measurements reported. The paper provides no quantitative measurement of the actual overhead: no GPU memory consumption comparisons between training with and without actor_attn, no throughput measurements (tokens per second or steps per hour), and no scaling analysis showing how the overhead grows with sequence length, model size, or batch size. The claim that overhead is "minimal" is unsupported.
Synchronization and infrastructure complexity. The three-model architecture (actor_infer, actor_attn, actor_train) requires weight synchronization after every policy update (Section 5.1). In distributed training environments, this means three sets of model weights must be kept consistent across potentially many GPUs and nodes. For actor_attn specifically, this involves loading updated weights, performing forward passes on the entire batch of generated responses, and communicating the resulting attention maps (or pre-computed γt values) to the training process. The communication overhead and engineering complexity of maintaining this synchronization are not discussed.
The cost is not amortized into the reported gains. The headline efficiency numbers (e.g., "coupled rhythm credit achieves 63.1%, substantially outperforming GRPO (52.6%)" on Countdown) are computed at identical training step counts (500 or 600 steps). If the additional memory overhead of actor_attn forces a reduction in batch size or an increase in gradient accumulation steps, the effective throughput of the credit-shaped training would be lower than GRPO at the same step count, reducing or eliminating the practical efficiency gain. The paper provides no such accounting.

What evidence exists in the paper. No experiments measure the computational cost of the attention extraction. The paper's entire empirical case for the method's efficiency rests on the comparison of training curves at fixed step counts (Figures 4, 5) and final performance metrics (Tables 3, 4), without any measurement of per-step wall-clock time, memory usage, or total FLOPs. The statement about "little additional latency" in Section 5.1 is asserted but not measured.

Mitigation status. The paper does not address this limitation. The method is presented as "plug-and-play with standard RLVR" (Section 6) without quantifying what "plug-and-play" costs in practice. A practitioner evaluating whether to adopt this method cannot determine whether the performance gains (e.g., +3.8 points average on 8B mathematical reasoning) justify the additional infrastructure complexity and potential throughput reduction without measurements that the paper does not provide. Future work could provide wall-clock timing comparisons, memory profiling, and scaling analysis to enable cost-benefit evaluation.

6.3 The Method Requires a Separate Eager-Attention Model Instance, Limiting Deployment in Memory-Constrained Settings

The assumption or constraint. The credit assignment mechanism fundamentally depends on accessing full attention matrices during training, which is incompatible with flash attention—the standard attention implementation in modern LLM training and inference frameworks (vLLM, Megatron, etc.). The paper's solution is a dedicated actor_attn model instance running standard (eager) attention, synchronized with the inference and training instances (Section 5.1). This means the training setup requires three copies of the model weights in memory simultaneously: one for inference (actor_infer, flash-attention-optimized), one for attention extraction (actor_attn, memory-intensive eager attention), and one for training (actor_train, flash-attention-optimized with optimizer states).

The consequence. This architecture imposes hardware requirements that may be prohibitive in resource-constrained settings:

GPU memory pressure. Training already strains GPU memory with model weights, optimizer states (typically 2-3× the model weights for Adam-style optimizers), and activations. Adding a third model instance—particularly one that must materialize full attention matrices for long sequences—significantly increases peak memory usage. For the 8B model with 8K context, the actor_attn instance alone requires substantial memory for the attention maps (at least five 8192×8192 matrices per batch element, depending on batch size). The paper's experiments use 8 GPUs for 4B training and 16 GPUs for 8B training (Section 5.3.1), but does not report whether this GPU count was chosen to accommodate the actor_attn memory overhead or whether GRPO training could have been done with fewer GPUs. A practitioner with fixed hardware may find that the method requires reducing batch size, sequence length, or model size to fit within memory constraints, potentially offsetting the performance gains.
Limited to training-time use. The credit assignment strategies are designed for RL training, not inference. The actor_attn instance and the attention extraction pipeline are not usable during deployment, meaning the method only affects the training process—the final deployed model has no special architecture or inference-time advantage. This is not a flaw per se (the paper never claims inference-time benefits), but it means the infrastructure cost is borne entirely during training, unlike methods that improve inference-time efficiency (e.g., model distillation, quantization). For organizations that train models infrequently but deploy them widely, the training overhead may be acceptable; for organizations that retrain frequently or operate with tight training budgets, the additional infrastructure cost may be harder to justify.
Engineering complexity for frameworks that do not support multi-instance architectures. The paper implements the method on top of the ROLL framework (Wang et al., 2025c), which is specifically designed for multi-instance RL training with separate inference and training engines. For research groups or practitioners using simpler training setups (e.g., a single-model training loop with on-policy generation), integrating the actor_attn instance and synchronization logic represents non-trivial engineering work that the paper does not provide as reusable infrastructure. The "plug-and-play" characterization (Section 6) overstates the ease of adoption for users not already using ROLL-style multi-instance architectures.

What evidence exists in the paper. The paper provides no hardware requirement comparison between GRPO training and credit-shaped training. The GPU counts (8 for 4B, 16 for 8B) are reported as the training configuration but not compared to the minimum GPU count needed for baseline GRPO training at the same batch size and sequence length. The paper does not report peak GPU memory usage, communication overhead between instances, or any failure modes encountered during the three-instance synchronization.

Mitigation status. The paper does not address this limitation. The architecture is presented as a solution to the flash attention problem (Section 5.1) without discussing its memory or engineering costs. The paper does not explore alternatives that might reduce the memory burden—for example, extracting attention from a subset of layers more aggressively, approximating WAAD/FAI from partial attention patterns, or using techniques to reconstruct attention statistics from flash attention's intermediate computations without materializing the full matrix. These are left as implicit future work.

6.4 The Back-Allocation Mechanism in Coupled Rhythm Credit Lacks an Isolating Ablation

The assumption or constraint. The coupled rhythm credit strategy (Strategy 3, Section 5.2) combines three distinct mechanisms: (1) amplification of preplan tokens identified by WAAD drops, (2) amplification of anchor tokens identified by high FAI, and (3) back-allocation of credit from locally dominated anchors to their associated preplan tokens (Equation 18). The strategy introduces the concept of locally dominated anchors—high-FAI tokens whose own WAAD is low (indicating local attention) and that immediately follow a WAAD peak—and uses the parameter α to control the fraction of the anchor's amplification bonus that is shifted to the preceding preplan token.

The consequence. Without an ablation that isolates the contribution of the back-allocation mechanism specifically, a practitioner cannot determine whether the performance advantage of coupled rhythm credit over local-chunk credit and global-anchor credit comes from:

The additive combination of preplan and anchor signals (i.e., amplifying both sets of tokens independently would achieve the same gain), or
The back-allocation mechanism specifically (i.e., the credit shift from locally dominated anchors to their preplans addresses a genuine optimization constraint that neither signal alone captures).

This distinction matters for both scientific understanding and practical deployment. If the gains come primarily from combining preplan and anchor amplification, then a simpler strategy—amplify both WAAD-detected preplan tokens AND high-FAI anchor tokens simultaneously, without any back-allocation—might achieve equivalent performance with less complexity (no need to detect locally dominated anchors, no need to tune α and k). If the gains depend specifically on back-allocation, then the locally dominated anchor concept represents a genuine insight about reasoning structure that practitioners must implement to achieve the best results.

The paper's results show that coupled rhythm credit consistently outperforms both local-chunk and global-anchor credit individually (e.g., on 8B-1K math: coupled 43.2% vs. global 42.1% vs. local 41.3%; Table 4), but this does not distinguish between the two mechanisms because coupled credit applies BOTH the combination AND the back-allocation. A direct ablation comparing coupled credit with and without back-allocation (α = 0 vs. α > 0, or equivalently, a "joint amplification" baseline that amplifies both preplan and anchor tokens without any credit shifting) would resolve this ambiguity.

What evidence exists in the paper. No experiment varies α (the back-allocation fraction) or compares coupled rhythm credit to a simple joint amplification baseline (amplify both WAAD-detected and FAI-detected tokens with no reallocation). The top-k ratio ablation (Figure 7, Table 5) varies the quantile threshold for coupled credit but keeps the back-allocation mechanism active. The bottom-k ablation (Table 5) tests whether credit assignment direction matters but does not test back-allocation specifically. The neighborhood size k ∈ {1, 2, 3} and thresholds τ_waad and τ_Δ are mentioned as hyperparameters (Section 5.3.1) but no sensitivity analysis is provided for these values.

Mitigation status. The paper does not address this limitation. The coupled rhythm credit strategy is presented as a unified method without decomposing its contributions, and the experiments do not provide evidence to distinguish between the combination hypothesis and the back-allocation hypothesis. A practitioner implementing the method cannot determine whether they need to implement the full locally-dominated-anchor detection and back-allocation pipeline, or whether a simpler joint amplification of WAAD-detected and FAI-detected tokens would suffice. The paper's innovation claim around the locally dominated anchor mechanism (Section 4.2: "this motivates joint consideration of both the anchor and its introductory token during optimization") rests on this unablated component.

6.5 Hard Problems Remain Essentially Unaddressed: No Evidence the Method Helps When the Base Model Lacks Capability

The assumption or constraint. The preplan-and-anchor rhythm and the associated credit assignment strategies are fundamentally mechanisms for amplifying and organizing reasoning that the model already possesses, not for creating new reasoning capabilities. If the base model cannot generate correct reasoning traces for a problem type—even with extensive sampling and search—no amount of credit assignment refinement can produce correct reasoning, because there is no correct trajectory to reinforce.

The consequence. This limitation manifests in two ways in the paper's results:

Performance ceilings across benchmarks. Despite consistent improvements over GRPO, the absolute performance of the credit-shaped models on the hardest benchmarks remains low. On AIME25 (the most challenging math benchmark), the best result is 12.3% for Qwen3-8B with coupled rhythm credit at 1K context (Table 4)—this is a +5.0 point gain over GRPO's 7.3%, but the model still fails on 87.7% of problems. On OlympiadBench, the best result is 47.0% at 1K and 52.2% at 8K, meaning roughly half of problems remain unsolved. These failure modes are not analyzed: the paper does not investigate whether the credit assignment strategies fail on the hardest problems (e.g., by misidentifying preplan/anchor tokens when the model's reasoning is fundamentally incorrect) or simply cannot overcome the base model's capability limitations.
No analysis of problem difficulty. Unlike some prior work on test-time compute scaling (e.g., the example paper's analysis of difficulty-conditioned optimal strategies), this paper does not bin problems by difficulty and report per-bin improvements. The aggregated results in Tables 3 and 4 obscure whether the gains are concentrated on easy/medium problems (where the base model already has some capacity to produce correct reasoning) and absent on hard problems (where it does not), or whether the gains are uniform across difficulty levels. Given the "hard problems remain essentially unsolved" finding from prior work on test-time compute scaling, it is plausible that the credit assignment strategies similarly provide no benefit on problems outside the base model's capability range—but the paper provides no evidence either way.

What evidence exists in the paper. The paper provides only aggregate performance metrics without difficulty-level breakdowns. The perturbation experiments (Section 4.1, Table 1) use 70 randomly sampled GSM8K problems but do not report whether the causal effect of high-FAI perturbation varies with problem difficulty. The training curves (Figures 4, 5) show aggregate performance over training steps without difficulty-conditioned analysis.

Mitigation status. The paper does not address this limitation. The method is presented as broadly beneficial for reasoning RL without characterizing its boundary conditions. A practitioner needs to know: if my model's pass@1 on a target benchmark is X%, what improvement can I reasonably expect from attention-derived credit assignment? Does the method help primarily by making the model more consistent on problems it can already sometimes solve (improving pass@k for small k), or does it help the model discover correct reasoning on problems it previously never solved? The paper provides no evidence to answer these questions. Difficulty-binned analysis—analogous to the difficulty quintile analysis in the compute-optimal test-time scaling paper—would substantially strengthen the practical guidance value of the results.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes a new axis of optimization for LLM reasoning—structure-aware credit assignment derived from the model's own attention dynamics—that sits between two previously disconnected research threads. On one side, mechanistic interpretability has produced rich descriptive accounts of how LLMs process information internally (iteration heads, induction heads, deduction circuits, attention sinks) but has largely stopped at diagnosis. On the other, reinforcement learning for reasoning has focused on reward design and algorithm variants (PPO, GRPO, DPO, PRM-guided search) while treating the distribution of credit across tokens as a fixed consequence of those choices, typically uniform. The paper closes this gap by demonstrating that attention-derived signals can be operationalized during training to differentially reinforce structurally decisive tokens, without requiring architectural changes or new loss functions.

The magnitude of this shift is best characterized as a methodological reframing with practical impact, rather than a paradigm shift. The paper does not propose a new model architecture, a new attention mechanism, or a fundamental revision to RL theory. It provides a template for converting interpretability insights into optimization interventions within existing RLVR pipelines. The template has three steps: (1) identify a structurally meaningful signal from attention dynamics, (2) validate its causal role through perturbation experiments, and (3) operationalize it as a multiplicative advantage scaling factor within standard PPO/GRPO objectives. This template is instantiated concretely through WAAD, FAI, and the three credit assignment strategies, but the conceptual contribution is the template itself—any attention-derived signal that captures structurally important positions (decision points, planning tokens, error-correction positions, backtracking points) could be similarly operationalized.

The paper's most significant reframing is repositioning interpretability as an input to optimization rather than merely an output of analysis. Prior to this work, the implicit logic of interpretability research was: first understand the model, then (maybe, eventually, at some undefined point in the future) use that understanding to improve it. This paper collapses that loop into a single training process: the model's self-analysis (reading its own attention patterns via actor_attn) directly shapes its self-improvement (scaling advantages during RL). This is a step toward reflexive training systems where the model's internal organization guides its own optimization, reducing dependence on external reward signals that are expensive to design, expensive to compute (in the case of PRMs), and susceptible to reward hacking.

The work also resolves a latent tension in the entropy-based exploration literature. Several recent papers (Wang et al., 2025b; Cheng et al., 2025; Cui et al., 2025) have identified high-entropy tokens as important for reasoning and have proposed entropy-modulated RL strategies that amplify advantages at uncertain positions to encourage exploration. The implicit claim in that line of work is that entropy is a sufficient proxy for token importance. This paper provides evidence that entropy is a symptom of an underlying structural process—the preplan phase of the preplan-and-anchor rhythm—and that directly measuring the structural process (via WAAD, which captures long-range context consultation) provides a cleaner signal than the behavioral proxy (entropy, which conflates genuine planning junctures with arbitrary stylistic uncertainty). The results in Tables 3 and 4 consistently show WAAD/FAI-based credit assignment outperforming entropy-based credit assignment (e.g., coupled rhythm credit at 63.1% vs. high-entropy credit at 57.7% on Countdown; 43.2% vs. 40.1% on 8B-1K math), validating that attention-derived signals capture information beyond what output uncertainty alone provides. This does not invalidate entropy-based approaches—entropy remains valuable as a computationally cheap proxy when attention extraction is infeasible—but it establishes that direct measurement of attention structure provides a superior signal when available.

The paper also makes the verifier-free credit assignment paradigm more attractive as a research direction. Process reward models (PRMs) represent the dominant approach to fine-grained credit assignment in reasoning RL, but they require expensive step-level supervision (human annotations or Monte Carlo rollouts), are susceptible to reward hacking as the policy improves, and introduce distribution shift between training and deployment. This paper demonstrates that the model's own attention provides a free, always-on-policy signal for identifying structurally important tokens, circumventing all three PRM limitations simultaneously. The perturbation experiments (Table 1) establish that this signal is causally meaningful—high-FAI tokens causally influence downstream reasoning—which addresses the concern that attention-based importance might be merely correlational. While the paper does not directly compare against PRM-based credit assignment, it establishes the viability of an alternative paradigm that may be preferable when step-level supervision is unavailable or when computational efficiency during training is a priority.

Research directions that become more attractive as a result of this work include: (1) attention-based credit assignment for any task where the model exhibits structured attention patterns (not limited to mathematical reasoning), (2) combining attention-derived signals with other importance indicators (entropy, PRM scores, gradient-based attribution) to create multi-signal credit assignment strategies, (3) using the preplan-and-anchor rhythm as a diagnostic for reasoning quality—does a model that exhibits stronger rhythmic structure produce more reliable reasoning?—and (4) developing hardware-efficient methods for attention extraction during training to reduce the memory overhead identified in Section 6.2.

Research directions that become less urgent include: (1) developing ever-more-complex PRM architectures for step-level scoring, if attention-derived signals can provide comparable fine-grained credit assignment without any learned verifier; (2) exploration strategies that rely solely on entropy-based token selection without considering the structural role of uncertain positions; and (3) purely behavioral approaches to identifying important reasoning tokens that ignore the model's internal attention organization, since the paper demonstrates that attention provides a structurally grounded signal that outperforms behavioral proxies.

Follow-Up Research This Work Enables

Direct comparison of attention-derived credit assignment against process reward model (PRM) baselines. The paper motivates its approach partly as an alternative to PRM-based fine-grained credit, but provides no direct comparison. A strong follow-up study would train GRPO with three credit assignment methods on the same base model and dataset: (1) uniform credit (baseline), (2) process reward model credit using a PRM trained via Monte Carlo rollouts (following Wang et al., 2024), and (3) attention-derived credit using coupled rhythm. This would answer the critical question: does the model-intrinsic signal from attention match or exceed the performance of a learned process verifier? Additional conditions could test complementary effects—for example, using PRM scores to determine whether a reasoning step is correct, and using WAAD/FAI to determine how to distribute credit within that step across preplan and anchor tokens. The Countdown puzzle and MATH500 would provide controlled testbeds with clean correctness signals. The key metrics would be final accuracy and training sample efficiency (performance vs. number of RL steps).

Replicating the attention dynamics analysis on non-math reasoning tasks at multiple model scales. The paper's mechanistic analysis is conducted on a single model (Qwen3-4B-Base) and a single task (GSM8K). To establish the preplan-and-anchor rhythm as a domain-general feature of LLM reasoning, a replication study would extract attention maps from Qwen3-8B-Base, Llama-3-8B, and a larger model (e.g., 14B or 32B parameters) on at least three task types: mathematical reasoning (GSM8K or MATH), code generation (HumanEval or MBPP), and multi-hop QA (HotpotQA). For each model-task combination, the study would compute WAAD, FAI, and their coupling statistics, and test whether the three coupling patterns (WAAD-entropy, receiver-global FAI concordance, WAAD-FAI temporal alignment) hold above random baselines with comparable lift magnitudes. A negative result—the rhythm appearing only in math reasoning or only at the 4B scale—would significantly bound the generality of the paper's claims and guide practitioners on when the method is applicable. A positive result—the rhythm appearing across tasks and scales—would establish it as a genuine reasoning primitive rather than a task-specific artifact.

Isolating the contribution of the back-allocation mechanism in coupled rhythm credit. The coupled rhythm credit strategy (Strategy 3) combines three elements: preplan amplification, anchor amplification, and back-allocation from locally dominated anchors to their preceding preplans. The paper does not ablate back-allocation separately, leaving unclear whether the performance advantage of coupled credit over local-chunk and global-anchor credit comes from combining preplan and anchor signals, or specifically from the back-allocation mechanism. A clean ablation would compare four conditions on Qwen3-8B-Base with 1K context on the math benchmarks: (a) local-chunk credit only, (b) global-anchor credit only, (c) joint amplification (amplify both WAAD-detected and FAI-detected tokens simultaneously with no reallocation, i.e., coupled credit with α = 0), and (d) coupled rhythm credit with back-allocation (α > 0). If condition (c) matches condition (d), the back-allocation mechanism is unnecessary and the locally dominated anchor concept does not contribute beyond simple signal combination. If condition (d) meaningfully exceeds (c), it validates the paper's claim that locally dominated anchors have limited optimization capacity and that credit should shift upstream. An additional sweep over α values (0.0, 0.25, 0.5, 0.75, 1.0) would characterize the sensitivity of this mechanism and identify the optimal reallocation fraction.

Data efficiency characterization—does attention-derived credit assignment reduce the number of training samples needed? The paper's motivation claims that structure-aware credit assignment should improve data efficiency, but all experiments use fixed training set sizes (20K for Countdown, 17K for math) and fixed training durations (500–600 steps). A data efficiency study would train GRPO and coupled rhythm credit on varying fractions of the DAPO-Math-17K dataset (e.g., 25%, 50%, 75%, 100%) and measure the performance at each data volume. The key question: does coupled rhythm credit achieve the same performance as GRPO with less training data, or achieve higher performance with the same data? If coupled credit at 50% data matches GRPO at 100% data, it would establish a 2× data efficiency improvement, which is a much stronger practical claim than the current fixed-data comparisons. The study should also measure whether the efficiency gain is larger for harder problems (where structural credit assignment might matter more because random exploration is less likely to find correct solutions) or easier problems.

Stress-testing the method on models and architectures beyond Qwen. The paper's experiments are confined to Qwen3 models. A necessary stress test for the generality of the approach would evaluate the three credit assignment strategies on a model with a substantially different architecture and pretraining distribution—for example, Llama-3-8B, Gemma-2-9B, or DeepSeek-Coder-7B. The study would first replicate the head classification and coupling pattern analysis on the target model to verify that local/global attention patterns and the preplan-and-anchor rhythm exist. If they do, the RL experiments would proceed with the same DAPO-Math-17K training data and the same Countdown and math benchmarks. A negative result—the credit assignment strategies providing no gain or negative gain on a non-Qwen model—would indicate that the preplan-and-anchor rhythm is architecture-specific or pretraining-data-specific, severely limiting the method's applicability. A positive result would strengthen the case that the rhythm reflects a general property of autoregressive transformer reasoning rather than a Qwen-specific implementation detail.

Developing and evaluating lightweight attention proxies to reduce memory overhead. The paper's requirement for eager attention extraction via a separate model instance is its primary practical limitation (Section 6.2). A follow-up engineering study would develop and test methods for estimating WAAD and FAI without materializing full attention matrices. Candidates include: (1) computing attention statistics from the flash attention intermediate computations without storing the full matrix (e.g., accumulating column-wise sums for FAI and row-wise expected distances for WAAD using streaming algorithms during the flash attention forward pass), (2) training a small probe network that predicts WAAD/FAI rankings from hidden states alone (avoiding attention extraction entirely), and (3) using attention maps from only one or two layers (rather than five) and testing whether the resulting γt values produce similar RL gains. The evaluation metric would be both the correlation between proxy WAAD/FAI and true WAAD/FAI, and the downstream RL performance using proxy-derived credit assignment vs. true attention-derived credit assignment. If a lightweight proxy achieves comparable RL performance, it would remove the primary barrier to practical adoption.

Combining attention-derived credit assignment with process reward models. The paper positions its approach as an alternative to learned verifiers, but the two sources of signal are potentially complementary—PRMs score what the model wrote (is this reasoning step correct?), while WAAD/FAI capture how the model organized its internal processing (was this token a preplan or anchor?). A study would test a hybrid credit assignment strategy where advantages are scaled by both a PRM-derived importance weight and an attention-derived structural weight: γt = γ_t^PRM × γ_t^attn (or a learned weighted combination). The experiment would compare four conditions on mathematical reasoning: GRPO uniform, attention-derived credit only, PRM credit only, and the hybrid. The key question is whether the signals are redundant (hybrid ≈ max(attention, PRM)), complementary (hybrid > max(attention, PRM)), or interfering (hybrid < max). The Countdown task—where the paper's attention-derived credit shows the largest gains (+10.5 points over GRPO)—would be a strong testbed because the attention signal is clearly effective there, and adding PRM signal might either boost further or introduce conflicting credit assignments.

Practical Applications and Downstream Use Cases

Improving data efficiency in RLVR training for reasoning models. The paper demonstrates that attention-derived credit assignment produces higher accuracy than standard GRPO at the same number of training steps (e.g., 43.2% vs. 39.4% average on 8B-1K math, Table 4). For organizations training reasoning models from scratch via RL, this translates directly to either higher final performance with the same training budget, or equivalent performance with fewer training steps. The Countdown results are particularly striking: coupled rhythm credit achieves 63.1% compared to GRPO's 52.6% after the same 500 training steps (Table 3). For a small model (4B parameters, 8 GPUs), the infrastructure overhead of adding an actor_attn instance with eager attention is manageable, and the 20% relative improvement in accuracy justifies the engineering cost. The primary adoption barrier is for teams not already using a multi-instance training architecture (separate inference, training, and now attention instances), where the engineering integration effort may exceed the benefit for small-scale experiments but becomes worthwhile at production scale.

Guiding exploration in self-improvement loops and iterative training pipelines. The paper's finding that high-FAI tokens causally govern downstream reasoning (Table 1: 87.14% of perturbation trials show greater trajectory divergence at high-FAI positions) suggests a concrete application beyond credit assignment: targeted data augmentation. In self-improvement pipelines where a model generates training data for its own fine-tuning (STaR, ReST^EM, rejection sampling), the attention-derived signals could identify positions where alternative token choices would most effectively diversify the training data. Specifically, at high-FAI preplan positions (where a WAAD peak coincides with or precedes a high-FAI anchor), forcing the model to sample from its top-k distribution rather than taking the argmax could generate semantically diverse training trajectories that explore different reasoning strategies, while perturbations at low-FAI positions could be suppressed to avoid generating surface-level variations that do not change reasoning structure. This would concentrate the generation budget on positions where exploration is most likely to produce genuinely different reasoning paths, improving the diversity-to-cost ratio of self-improvement data generation.

Diagnosing and improving the reasoning quality of deployed models without retraining. The preplan-and-anchor rhythm provides a diagnostic lens that can be applied to any model at inference time (a single forward pass suffices for attention extraction, as the paper does with actor_attn). For a deployed reasoning model, a monitoring system could compute WAAD and FAI on generated responses to specific query types and track metrics over time: the strength of the WAAD- entropy coupling, the density of high-FAI anchor tokens, and the WAAD-FAI temporal alignment. Degradation in these metrics could indicate that the model's reasoning structure is becoming less organized (e.g., due to distribution shift in user queries, or after a model update), potentially flagging problems before they manifest as accuracy drops in user-facing metrics. This application does not require the RL training infrastructure—only the attention extraction pipeline—making it more immediately deployable than the full credit assignment method.

Structured credit assignment for tasks where verifiable rewards are unavailable or expensive. The paper's credit assignment mechanism requires only a sequence-level reward (the same reward already used in RLVR) and the model's own attention maps—no step-level supervision, no learned process verifier, no human annotation of intermediate reasoning steps. This makes it applicable to reasoning tasks where ground-truth intermediate steps are unavailable or correctness is ambiguous: open-ended essay writing, creative problem-solving, strategic planning, or multi-step decision-making in environments where only final outcomes are scored. For example, in an agentic task where the model generates a multi-step plan and receives only a binary success/failure reward at the end, uniform credit assignment would spread that binary signal across all tokens equally. Attention-derived credit assignment would automatically concentrate the signal on the preplan and anchor tokens—the positions where the model commits to specific sub-goals and intermediate results—potentially accelerating learning without requiring a human to annotate which planning steps were good or bad. The paper does not test this on agentic tasks, but the conceptual framework (WAAD and FAI as domain-general attention patterns rather than math-specific metrics) supports the extension, and the consistent gains across Countdown puzzles, QA, and math (Tables 3, 4) suggest task-type robustness.