Large Reasoning Models Learn Better Alignment from Flawed Thinking

🎯 Pitch

Large reasoning models can be taught to override flawed safety reasoning by training them on deliberately misaligned chain-of-thought prefills, a counterintuitive strategy that improves jailbreak robustness by over 20% while preserving mathematical reasoning—yet the benefit vanishes if the training traces are already safe.

1. Executive Summary

This paper proposes RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), a principled reinforcement learning method for post-training large reasoning models to override flawed reasoning trajectories and reroute to safe and helpful responses. The method trains on a mixture of synthetically generated counter-aligned chain-of-thought prefills—flawed reasoning that induces unsafe thinking on harmful prompts (e.g., prefilling with traces from a weakly safety-aligned model) and overly conservative thinking on benign prompts (e.g., prefilling with traces from a refusal-prone model)—alongside standard prompts, requiring no additional training cost or pipeline modifications beyond vanilla RLHF. Evaluated on DeepSeek-distilled Llama-8B and Qwen-14B across safety (StrongREJECT, WildJailbreak, Fortress), overrefusal (XSTest), and mathematical reasoning benchmarks (MATH500, GSM8K, AIME2024), RECAP achieves on average +12.3% on direct harmful benchmarks, +21.0% on jailbreaking benchmarks, and +7.8% helpfulness on overrefusal, while preserving math reasoning capability and maintaining the same inference-time token budget as vanilla RLHF—establishing that training with counter-aligned reasoning prefills induces persistent self-reflection and robustness under adaptive attacks only when the model is compelled to override flawed trajectories rather than exploit already-safe reasoning traces.

2. Context and Motivation

The Core Problem: Reasoning Models Follow Instead of Think

The fundamental tension this paper addresses is deceptively simple yet profound for the field of AI safety: large reasoning models exhibit emergent "thinking" behaviors during chain-of-thought generation, yet they remain unable to reason critically about safety alignment. When a flawed premise is injected into their reasoning trace—even one they could recognize as problematic if generating from scratch—they tend to follow it rather than override it. This brittleness means that the same models demonstrating impressive "aha moments" during mathematical reasoning (Guo et al., 2025), where they revisit and refine their own steps, fail to apply analogous self-correction when the reasoning trajectory veers into unsafe territory.

This gap matters for several critical reasons the authors establish in Section 1 and Section 2:

CoT prefilling is widely supported in production systems. Both open-source models (Jeung et al., 2025) and commercial APIs (Anthropic, 2025) support CoT prefilling, where a developer or user can inject a partial reasoning trace before the model generates its continuation. This feature is intentionally designed—it provides control over output structure—but as the paper demonstrates, it creates a direct attack surface: a malicious actor (or even a careless user) can prefill with "I know that" or more elaborate unsafe reasoning to bypass alignment constraints entirely (Rager et al., 2025). The vulnerability is not theoretical; it is a present-day risk in deployed systems.
The training-inference mismatch for reasoning models. During online RL training, LRMs are commonly rewarded only for correct final responses—the optimizer sees (prompt, final_answer, reward) without examining the quality of intermediate reasoning steps (Lambert et al., 2024). At inference time, however, the model must navigate through its own noisy reasoning trajectories, which may begin with flawed or misleading steps. There is a generalization gap: the model has never been trained to recover from bad reasoning because during training, its rollouts start from an unconditioned initial state. The emergence of self-correction during mathematical reasoning (the celebrated "aha moments" in DeepSeek-R1) is a happy accident of the training process for capability tasks, but it does not reliably transfer to safety-critical contexts where the stakes are higher and the reward signal during training may be sparser.
Safety alignment for reasoning models is understudied relative to their rapid deployment. Frontier LRMs—DeepSeek-R1 (Guo et al., 2025), OpenAI-o3 (OpenAI), Qwen3 (Team, 2025)—have achieved remarkable benchmarks in math (Shao et al., 2024) and coding (Jiang et al., 2024a), driving their adoption in open-ended applications. Yet as the paper notes in Section 6, concurrent work has revealed that these models exhibit substantial safety vulnerabilities (Knight et al., 2025; Jiang et al., 2025; Wang et al., 2025b; Zhou et al., 2025b; Arrieta et al., 2025). The tension between capability and safety is particularly acute because the very mechanism that makes LRMs powerful—long, structured CoT reasoning—also creates a larger attack surface: more tokens of internally generated content means more opportunities for unsafe reasoning to propagate.

Section 2's Controlled Demonstration: Quantifying the Brittleness

The paper does not simply assert that LRMs are brittle; Section 2 constructs a controlled empirical demonstration that makes the problem concrete and measurable. This is essential context for understanding why RECAP's training recipe is designed the way it is.

The experimental design is straightforward but incisive. The authors take a set of DeepSeek-distilled LRMs spanning model sizes from 1.5B to 32B parameters (DSQwen-1.5B through DSQwen-32B, plus DSLlama-8B and DSQwen-14B) and evaluate their safety on the StrongREJECT benchmark (313 harmful prompts; Souly et al., 2024), where the final response is judged by GPT-4o and reported as a "safety score" (percentage of completions judged safe). They then inject the first 200 words of chain-of-thought reasoning from other models into each model's CoT and measure how safety scores shift.

Finding 1: Unsafe prefills drag safety down dramatically. When models are prefilled with the first 200 words of CoT from DSQwen-1.5B—the least safe model in the lineup—the average safety score across the four larger DS models drops by 36.4% (Table 1, ypre_cot from DSQwen-1.5B row). For example, DSQwen-32B drops from an original safety score of 72.20 to 35.14 when forced to continue from the 1.5B model's unsafe reasoning. This means a model that otherwise produces safe responses over 70% of the time becomes unsafe more than 60% of the time simply because its reasoning trace is seeded with flawed premises.

Finding 2: Safe prefills boost safety equally dramatically. The inverse also holds: when prefilled with CoT from the safest model (DSQwen-32B, original score 72.20), the four smaller DS models see their average safety score increase by 91.7% (Table 1). This symmetry—flawed reasoning degrades behavior as much as good reasoning improves it—is the smoking gun that LRMs are not performing independent critical evaluation of their reasoning traces; they are inheriting the quality (or lack thereof) of whatever reasoning they are seeded with.

Finding 3: The brittleness crosses model families. Crucially, the paper tests Qwen3-4B-Thinking-2507 (Team, 2025), a reasoning model from the Qwen 3.0 family trained with a different dataset and recipe than the DSQwen models. Qwen3-4B achieves a high safety score of 95.21 under standard evaluation, but drops by 19.5% when prefilled with DSQwen-1.5B's CoT and by 11.4% when prefilled with DSQwen-32B's CoT (Table 1, bottom rows). The fact that brittleness persists across model families, training recipes, and data distributions indicates this is not a quirk of one particular training run but a structural vulnerability of the current LRM training paradigm.

Finding 4: The brittleness generalizes beyond safety. Appendix B extends the controlled study to mathematical reasoning (MATH500) and overrefusal (XSTest). On MATH500, weak prefills from DSQwen-1.5B reduce accuracy across all models; strong prefills from DSQwen-32B boost it, with the effect being remarkably consistent—DSQwen-14B, for instance, drops from 86.40 to 82.60 with weak prefills and rises to 92.40 with strong prefills (Table 5). On XSTest, Qwen3-4B's helpfulness score shoots from 84.0 to 93.2 when prefilled with DSQwen-32B's helpful reasoning (Table 6). This pattern—models inherit the quality of injected reasoning rather than reassessing it—is the paper's central diagnosis: LRMs follow without thinking, across domains and model families.

Where Existing Alignment Approaches Fall Short

The paper positions RECAP against a landscape of prior and concurrent work, identifying specific limitations that motivate the counter-aligned prefilling approach:

SFT on safety datasets teaches what to think, not how to recover. Methods like STAR (Wang et al., 2025b) and SafeChain (Jiang et al., 2025) construct reasoning-aligned datasets and apply supervised fine-tuning. STAR-1, for instance, uses just 1K examples to teach models to follow safe reasoning chains. These approaches can improve safety on in-distribution prompts—Table 2 shows STAR lifting DSLlama-8B from 57.83 to 77.00 on StrongREJECT—but they suffer from a fundamental limitation: they only demonstrate correct reasoning trajectories. The model learns to produce safe reasoning when generating from scratch, but never encounters situations where it must recognize and override incorrect reasoning. As the controlled study in Section 2 shows, once an adversary injects a flawed trajectory, the model lacks the skill to recover because it has never been trained to do so. Moreover, these SFT methods often increase overrefusal as a side effect—STAR drops DSLlama-8B's helpfulness on XSTest from 90.40 to 78.40 (Table 2)—because teaching a model to "be safe" without teaching it to distinguish safe from unsafe queries encourages blanket refusal.

Standard RLHF optimizes final responses, not reasoning trajectories. The dominant post-training paradigm—RLHF with algorithms like GRPO (Shao et al., 2024; Guo et al., 2025) or DAPO (Yu et al., 2025)—rewards models based on the quality of the final response y_resp, with the chain-of-thought y_cot treated as an unexamined intermediate computation. The objective is:

$\mathcal{J}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)}[R(x, y_\text{resp})]$

The CoT tokens are generated but not directly supervised. This works well for capability tasks where the CoT naturally converges toward correct reasoning (the reward signal for correct final answers implicitly shapes CoT quality), but it creates a vulnerability for safety: the model can learn to produce safe final responses while harboring unsafe intermediate reasoning, and it never practices the skill of noticing and correcting unsafe reasoning mid-trajectory. The paper's baseline DAPO results in Table 2 illustrate the consequences: while DAPO substantially improves safety over the original model (96.81 vs. 57.83 on StrongREJECT for DSLlama-8B), it still exhibits the brittleness documented in Section 2 when confronted with counter-aligned prefills. The safety gains are superficial—they reflect the model learning to start safe, not learning to stay safe.

Inference-time guardrails are patches, not solutions. Another line of work applies external safety filters (Inan et al., 2023) or runtime interventions that encourage reflection at inference time (Zhang et al., 2024). The paper acknowledges these approaches in Section 6 and Appendix I, but distinguishes RECAP by noting that inference-time methods "rely on runtime interventions" that add latency, complexity, and potential failure modes. They do not change the model's underlying inability to critically evaluate reasoning; they work around it.

Loss-modification approaches penalize unsafe reasoning but may not induce recovery. Recent work introduces auxiliary loss terms that shape the CoT directly—penalizing unsafe intermediate steps or rewarding safe trajectories (Mou et al., 2025; Jeung et al., 2025; Zhang et al., 2025c; Zhou et al., 2025c). While these methods improve safety metrics, the paper's ablation in Section 5.2 (Figure 4C) suggests a deeper issue: when models are given aligned CoT prefills during training (safe reasoning traces), they simply exploit them to achieve high rewards without learning to correct unsafe trajectories, and actually underperform vanilla DAPO on safety benchmarks. The mechanism matters: it is not enough to expose the model to safe reasoning; it must be forced to actively override flawed reasoning to build robust recovery behavior.

The Conceptual Gap This Paper Fills

The paper frames its contribution through a specific conceptual lens in the introduction:

"This raises a central question: How can we train LRMs to achieve robust safety alignment by recovering from misleading reasoning, rather than hoping self-correction will emerge implicitly?"

This question is the paper's intellectual anchor. It distinguishes RECAP from prior work along two axes:

Training objective vs. emergent behavior. Prior work has largely relied on self-correction as an emergent property of RL training for capability tasks—the "aha moment" is observed and celebrated, but not explicitly trained for. RECAP makes recovery from flawed reasoning the training objective itself by constructing counter-aligned prefills that force the model to override bad trajectories to achieve high rewards. This shifts the paradigm from "hope self-correction emerges" to "train self-correction explicitly."
In-distribution alignment vs. out-of-distribution robustness. Most prior safety alignment methods improve performance on the specific distribution of prompts and reasoning patterns seen during training. RECAP's counter-aligned prefilling is specifically designed to create a distribution shift during training — the model encounters reasoning trajectories from other models (DSQwen-7B for training, evaluated against DSQwen-1.5B prefills at test time, as noted in Section 5.1) — that forces it to learn a generalizable recovery skill rather than memorizing safe patterns. The paper's training dynamics analysis (Figure 3) confirms this: RECAP generalizes across prefill sources, maintaining high safety even when attacked with prefills from a model it never saw during training.

How RECAP Positions Itself

The paper does not claim to propose a fundamentally new RL algorithm or architecture. Instead, it positions RECAP as a training recipe modification that can be dropped into existing RLHF pipelines with zero additional training cost or pipeline changes:

"RECAP is easy to adopt, requiring no additional RL optimization steps or pipeline modifications."

This is an important framing choice. Rather than competing with DAPO, GRPO, or other policy optimization variants, RECAP augments them. The paper demonstrates this by showing RECAP works with both DAPO (main results, Table 2) and GRPO (Appendix D, Table 8), and with both continuous (logit-based) and binary reward signals (Appendix D, Table 7). The approach is agnostic to the underlying optimizer—it modifies what the model sees during training (counter-aligned reasoning traces) rather than how the optimizer updates parameters.

This positions RECAP as an alignment strategy rather than an optimization technique, making it complementary to advances in RL algorithms, reward modeling, and verifier design. A team already using DAPO for capability post-training can adopt RECAP by modifying their rollout generation procedure to include counter-aligned prefills on a fraction of prompts, with no changes to the training loop itself.

The paper also carves out a clear niche relative to the broader safety landscape. It does not aim to improve reward model quality (it uses off-the-shelf Granite-Guardian-3.1-8B and Llama-3.1-8B-Instruct judges), does not propose new prompt-level defenses or guardrails, and does not modify inference-time procedures. Its domain is purely in the training data distribution—specifically, the distribution of CoT prefixes the model encounters during online RL—and its claim is that systematic exposure to counter-aligned prefixes induces a qualitatively different reasoning behavior (persistent self-reflection, as documented in Section 5.3) that generalizes beyond the training distribution.

3. Technical Approach

3.1 Reader Orientation

RECAP is a training recipe—not a new model architecture or RL algorithm—that modifies which chain-of-thought prefixes a large reasoning model sees during online reinforcement learning. Instead of always generating rollouts from scratch, RECAP deliberately injects syntactically fluent but semantically misaligned ("counter-aligned") reasoning traces into a fraction of training prompts, forcing the model to override these flawed trajectories to achieve high rewards. The problem it solves is the brittleness documented in Section 2: LRMs follow injected reasoning rather than critically evaluating it, so RECAP turns this vulnerability into a training signal by making the model practice recovery during RL, thereby learning a generalizable self-correction behavior that transfers to standard, non-prefilled prompts at deployment.

3.2 Big-Picture Architecture (Diagram in Words)

The RECAP system has four major components that operate within a standard online RLHF loop:

A base large reasoning model πθ (e.g., DSLlama-8B or DSQwen-14B, DeepSeek-distilled models from Guo et al., 2025) that generates structured chain-of-thought reasoning ycot followed by a final response yresp. This is the policy being trained.
A counter-aligned prefill generator, which is an external misaligned model (or set of models) sampled during training to produce flawed CoT prefixes ypre_cot. For harmful prompts, the generator is a weakly safety-aligned or helpful-only model πharm that produces reasoning leading toward unsafe completions. For benign prompts, the generator is an overly conservative model πrefuse that produces reasoning leading toward refusals. These generators are frozen and sampled offline; they are not trained.
The RL policy optimizer (DAPO by default, Yu et al., 2025, but also demonstrated with GRPO in Appendix D), which computes advantages and updates the policy πθ based on scalar rewards assigned to the final response yresp. The optimizer operates identically to standard RLHF except that when a prompt is prefilled, the importance sampling ratio and advantage normalization skip the injected prefix tokens (they are treated as part of the "state" rather than as actions taken by the current policy).
A reward model suite providing multiple reward signals: IBM Granite-Guardian-3.1-8B (Padhi et al., 2024) for safety (using continuous logit-based rewards for denser feedback), a rubric-based overrefusal judge backed by Llama-3.1-8B-Instruct (Dubey et al., 2024) for helpfulness, and a Reinforcement Learning with Verifiable Rewards (RLVR) module (Lambert et al., 2024) that provides binary rewards for verifiably correct math answers.

Information flow during one RL update: A prompt x is sampled from the training corpus (5K prompts total: 1K harmful from BeaverTails, 1K overrefusal from STAR-1, 3K math from GSM8K and MATH). With probability α, a counter-aligned CoT prefix ypre_cot of length ℓpre words is injected before the model begins generating. The model πθ_old generates a group of G = 16 continuations ygen_cot followed by final responses yresp. A scalar reward Ri is computed for each rollout based on yresp only, ignoring ycot quality. Advantages are normalized within the group, and the policy is updated using the DAPO clipped objective, with tokens before ℓpre excluded from the importance sampling ratio. This is repeated for one epoch over the 5K training prompts. At deployment, no prefilling is applied—the model generates CoT from scratch, but the recovery behavior learned during training transfers.

3.3 Roadmap for the Deep Dive

First, the formal notation and definitions that underpin the entire method: how the LRM's output is decomposed into CoT and response, how prefilling is represented, and what "counter-aligned" means operationally. This establishes a shared vocabulary.
Second, the construction of counter-aligned prefills themselves—which generator models are used, how their reasoning traces are extracted, and the critical design choice of why the prefills must be semantically misaligned (unsafe for harmful prompts, refusal-oriented for benign prompts) rather than simply random noise or already-safe reasoning.
Third, how these prefills are integrated into the RLHF training loop: the fraction α of prompts that get prefilled, the length ℓpre of injected traces, the modified DAPO objective that excludes prefill tokens from policy optimization, and the dynamic sampling constraints that prevent degenerate training.
Fourth, the reward design that makes the training signal work: the specific reward models, their output types (continuous logits vs. binary labels), how multiple reward signals (safety, overrefusal, math) are jointly optimized, and why continuous rewards from Granite-Guardian provide denser gradients than thresholded binary labels.
Fifth, the training data composition and practical implementation details: dataset sizes, model scale, infrastructure, hyperparameters, and the crucial choices that make RECAP work without requiring additional RL optimization steps.
Sixth, the design rationale for why this particular combination—counter-aligned prefills + standard RLHF with prefix-masked optimization—induces persistent self-reflection where alternative approaches (aligned prefills, SFT on safety data, inference-time guardrails) fail.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that counter-aligned chain-of-thought prefilling during online RL teaches models to override flawed reasoning trajectories, building a generalizable recovery skill that standard training (which sees only clean, self-generated rollouts) fails to impart.

Notation and Formal Definitions

The paper defines a large reasoning model as a policy πθ parameterized by weights θ that, given an input prompt x, generates a complete output y = (ycot, yresp), where ycot is the intermediate chain-of-thought reasoning and yresp is the final response. This decomposition is standard for LRMs—models like DeepSeek-R1 and its distilled variants are trained to produce structured reasoning in a <think> block (or equivalent) before their final answer.

The key operation in RECAP is prefilling. A partial, pre-generated reasoning trace ypre_cot is injected before the model begins its own generation. Formally, the model receives the concatenated input (x, ypre_cot) and is asked to generate a continuation ygen_cot followed by yresp. The full output is:

$y = (y_{\text{pre}} \parallel y_{\text{gen}}, y_{\text{resp}})$

where $y_{\text{pre}}$ is the injected prefix, $y_{\text{gen}}$ is the model's own continuation of the chain-of-thought, $y_{\text{resp}}$ is the final response, $\parallel$ denotes sequence concatenation, and the complete CoT is $y_{\text{cot}} = y_{\text{pre}} \parallel y_{\text{gen}}$ .

What this notation describes operationally: at inference (or during a training rollout), the system takes a prompt x, retrieves a pre-generated CoT prefix ypre_cot from an external model, concatenates them, and feeds the result to the policy model as input. The policy then auto-regressively generates ygen_cot token-by-token, followed by yresp token-by-token. During standard (non-prefilled) generation, ypre_cot is simply empty and the model generates the entire CoT from scratch.

Why this decomposition matters for training: the scalar reward Ri is computed based only on the final response (x, yresp). The chain-of-thought—whether prefilled or model-generated—is invisible to the reward model. This means that during prefilled rollouts, the model could achieve high reward by simply following the flawed ypre_cot to an unsafe or refusal response. To actually achieve high reward, the model must override the flawed prefix—generating ygen_cot that corrects or reroutes the reasoning—and then produce an aligned yresp. The entire training objective is structured so that high reward is only attainable through this override behavior.

Constructing Counter-Aligned Prefills

The core technical contribution is in how the prefill traces ypre_cot are generated, not just that they are generated. The paper's key insight is that the prefills must be counter-aligned: they must push the model in the wrong direction for the given prompt type, creating a tension between following the prefix (which is syntactically fluent and semantically coherent) and achieving high reward (which requires overriding the prefix).

For harmful prompts (from the 1K BeaverTails subset): ypre_cot is sampled from a weakly safety-aligned or helpful-only model, denoted πharm. The paper uses DSQwen-7B as the generator for the main experiments (Table 2), but the ablation in Figure 4C also tests DSQwen-1.5B. These models are chosen because they produce reasoning that leads toward unsafe completions—for instance, reasoning that entertains the harmful query as legitimate, breaks down steps for executing harmful instructions, or fails to recognize ethical boundaries. The generator models are not the same as the policy being trained; DSQwen-7B is a distinct model from both DSLlama-8B and DSQwen-14B. This cross-model design is deliberate: it tests whether the learned recovery behavior generalizes to reasoning patterns the policy model has never seen during its own generation, and Section 5.1 confirms that RECAP trained with DSQwen-7B prefills generalizes to attacks using DSQwen-1.5B prefills.

For benign prompts (from the 1K STAR-1 subset that test overrefusal): ypre_cot is sampled from an overly conservative model πrefuse. The paper constructs this generator by first performing supervised fine-tuning (SFT) of DSQwen-7B on the STAR-1 safety-only subset, which contains primarily harmful prompts paired with rule-following chain-of-thought that often leads to refusals. When this SFT'd model is then applied to benign prompts from the overrefusal dataset, it produces reasoning oriented toward refusing the request—for example, reasoning that misinterprets a benign query as potentially harmful or that defaults to refusal as a precautionary stance. The paper explicitly notes this construction in Section 4.2:

"To construct counter-aligned traces for overrefusal, we first SFT DSQwen-7B on the STAR-1 safety-only subset, which primarily contains harmful prompts and rule-following CoT that often lead to refusals. We then apply this tuned model to the overrefusal dataset and extract its refusal-oriented reasoning traces as ypre_cot."

Why not use random noise or scrambled reasoning? The prefills must be syntactically fluent and semantically coherent—they must look like plausible reasoning that a model might actually generate. If the prefills were nonsensical or random, the model could learn to ignore them trivially (by detecting distribution shift) rather than learning to override them semantically. The counter-aligned property means the reasoning is wrong but plausible, forcing the model to engage with its content and actively decide to correct it.

Why not use aligned (safe) prefills? The ablation in Section 5.2 (Figure 4C) directly tests this alternative: when prefills are sourced from the STAR-1 safety subset—which pairs harmful prompts with safe reasoning—the trained model significantly underperforms vanilla DAPO on safety benchmarks. The paper's explanation is that safe prefills allow the model to "exploit them to achieve high reward, without learning to correct unsafe trajectories." The model simply continues the safe reasoning and produces a safe response, receiving positive reward without ever practicing the override skill. This is a critical design insight: counter-alignment is necessary, not incidental. The model must encounter a genuine conflict between the prefix and the reward to learn recovery.

The prefill length ℓpre is measured in words—not tokens—and the paper sweeps values from 100 to 700 words (Figure 4B), selecting ℓpre = 500 as the default for both safety and overrefusal training. At this length, the prefill provides a substantial reasoning trajectory (approximately several hundred tokens, depending on tokenization) that firmly establishes a flawed direction, giving the model enough context to need to actively override it. Shorter prefills (100–300 words) still outperform vanilla DAPO but provide weaker corrective supervision; excessively long prefills (700 words) cause performance decline, hypothesized as the model over-relying on the injected reasoning.

Integrating Prefills into the RLHF Training Loop

RECAP is implemented within the DAPO framework (Yu et al., 2025), an enhanced variant of Group Relative Policy Optimization (GRPO; Shao et al., 2024). The standard DAPO objective optimizes a policy by sampling groups of rollouts, computing advantages within each group, and applying a clipped importance sampling update. RECAP extends this with one key modification: prefix-aware importance sampling.

The training objective is:

$\mathcal{J}_{\text{RECAP}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}_{\text{prefill}}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{1}{\sum_{i=1}^G |o_i|_{\text{opt}}} \sum_{i=1}^G \sum_{t=t_0(x)}^{|o_i|} \min\left( r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}) \hat{A}_{i,t} \right) \right]$

subject to:

$|\{ o_i \mid R_i > \tau \}| < G \quad \text{and} \quad |\{ o_i \mid R_i = 0 \}| < G$

where:

$G = 16$ is the number of rollouts sampled per prompt (a "group")
$o_i$ is the $i$ -th rollout, consisting of all generated tokens (CoT continuation + response)
$|o_i|$ is the total length (number of tokens) of rollout $o_i$
$|o_i|_{\text{opt}} = |o_i| - (t_0(x) - 1)$ is the number of optimized tokens—those for which the policy is updated
$t_0(x) = 1$ if prompt $x$ is not prefilled, and $t_0(x) = \ell_{\text{pre}} + 1$ otherwise, where $\ell_{\text{pre}}$ is the number of tokens in the injected prefix (not words—the implementation converts the word-length $\ell_{\text{pre}}$ to token count automatically)
$r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid x, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid x, o_{i,<t})}$ is the per-token importance sampling ratio at timestep $t$
$\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}$ is the group-normalized advantage for rollout $i$ , broadcast to all tokens in that rollout
$\varepsilon_{\text{low}} = 0.2$ and $\varepsilon_{\text{high}} = 0.28$ are asymmetric clipping thresholds (following DAPO's convention of allowing more positive than negative deviation)
$\tau$ is a reward threshold used for dynamic sampling: entire prompt groups are discarded if all $G$ rollouts are bad ( $R_i = 0$ for all $i$ ) or uniformly good ( $R_i > \tau$ for all $i$ ), preventing wasted computation on prompts that are too hard or too easy

What this objective computes: for each batch of prompts, the system samples $G = 16$ rollouts from the old policy. Each rollout receives a scalar reward $R_i$ based solely on the final response $y_\text{resp}$ . Rewards within the group are standardized to mean zero, unit variance advantages $\hat{A}_{i,t}$ . The policy is updated by maximizing a clipped surrogate objective: for each optimized token (those at positions $t \geq t_0(x)$ ), the update increases the probability of tokens in high-advantage rollouts and decreases the probability of tokens in low-advantage rollouts, with clipping preventing overly large updates. The $\min$ operator implements pessimistic clipping—if the importance ratio $r_{i,t}$ moves outside $[1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}]$ , the gradient is zeroed for that token to prevent destructive updates.

Why the prefix-aware mask $t_0(x)$ is critical: without it, the policy would be updated on the prefill tokens $y_\text{pre}$ as if they were actions taken by the current policy, which is incorrect—those tokens were generated by a different model (the counter-aligned generator) and injected into the input. Optimizing over them would (a) give the policy a false credit/blame signal for tokens it didn't generate, and (b) waste optimization budget on tokens whose probability under $\pi_\theta$ may be essentially random relative to the actual generator. By starting optimization at $t_0(x) = \ell_{\text{pre}} + 1$ , the policy is only updated on tokens it actually generated—the CoT continuation $y_\text{gen}$ and the final response $y_\text{resp}$ . This means the model learns to produce $y_\text{gen}$ that routes away from the flawed $y_\text{pre}$ (since that continuation increases the probability of high-reward $y_\text{resp}$ ), without being directly penalized for the prefilled tokens themselves.

Why the dynamic sampling constraints matter: the two constraints $|\{o_i \mid R_i > \tau\}| < G$ and $|\{o_i \mid R_i = 0\}| < G$ implement curriculum learning by filtering. If a prompt is so hard that all 16 rollouts score zero (for safety: all unsafe; for math: all incorrect), the prompt provides no contrastive signal—every rollout has the same advantage (zero), and the update is vacuous. Similarly, if a prompt is so easy that all rollouts exceed the threshold $\tau$ , there's no differentiation between good and bad generations—again, advantages would be zero. Filtering these prompts ensures the training signal comes from prompts where the model produces both good and bad rollouts, providing meaningful gradient information. The threshold $\tau$ is not explicitly specified in the paper but is inherited from the DAPO framework's defaults.

The prefill fraction α: only a fraction $α \in (0, 1)$ of training prompts in the safety and overrefusal subsets receive counter-aligned prefills. The remaining $1 - α$ prompts are presented without prefilling (the model generates its entire CoT from scratch), and math prompts never receive prefills (to avoid degrading mathematical reasoning). The training distribution is therefore:

$\mathcal{D}_{\text{prefill}} = \{ x_i \in \mathcal{D}_{\text{train}} \mid \Pr(\text{augment } x_i \text{ with } y_{\text{pre}}) = α \}$

where $α = 0.5$ is the default used for all main experiments.

Why α < 1 is necessary: the ablation in Section 5.2 (Figure 4A) shows that $α = 1$ (all safety prompts prefilled) degrades performance relative to $α = 0.5$ . The paper's interpretation is that when all safety prompts are prefilled, "the model fails to learn how to initiate safe reasoning on its own, relying instead on the flawed prefixes." With $α = 0.5$ , the model sees both prefilled and non-prefilled prompts for the same task type, learning both (a) how to generate safe reasoning from scratch (on the non-prefilled prompts) and (b) how to override flawed reasoning when it encounters it (on the prefilled prompts). This mixture is essential—the model needs to practice both skills to generalize to deployment, where no prefilling occurs.

Math prompts and the absence of prefilling: crucially, the 3K math prompts from GSM8K and MATH never receive CoT prefilling during RECAP training. The paper states this explicitly in Section 4.2: "no CoT prefilling is applied to math prompts during training." The observed preservation (and slight improvement) of math performance is therefore an emergent property—the model's improved reasoning structure and self-reflection behavior, learned from the alignment-oriented prefilling, transfers positively to math without any direct math-prefill training. This is a key robustness result: RECAP does not trade off safety for capability; the skills are complementary.

Reward Design: Multi-Objective Optimization with Continuous and Binary Signals

RECAP optimizes three reward signals simultaneously, addressing safety, helpfulness (anti-overrefusal), and mathematical reasoning:

Safety reward: IBM Granite-Guardian-3.1-8B (Padhi et al., 2024), selected because it "ranks highest on the GuardBench leaderboard" (Bassani and Sanchez, 2024). The paper takes the model's logits as continuous reward signals rather than thresholded binary classification labels:

$R_{\text{safety}} = \text{logit}_{\text{safe}}(x, y_{\text{resp}})$

where $\text{logit}_{\text{safe}}$ is the raw (pre-softmax) output of the guardrail model corresponding to the "safe" class.

What this means operationally: for each generated response $y_\text{resp}$ to a harmful prompt, Granite-Guardian produces a logit value—a real number that, after softmax, becomes the probability of the response being safe. Using the raw logit rather than the 0/1 classification label provides denser gradient information: responses that are "almost safe" receive higher logits than responses that are "clearly unsafe," even if both would be classified as unsafe under a binary threshold. This density matters for RL training because it provides a meaningful advantage signal even when the model hasn't yet learned to produce fully safe responses—small improvements in safety quality translate to higher rewards, creating a smoother optimization landscape.

Why continuous over binary: Appendix D (Table 7) compares RECAP trained with binary Granite-Guardian labels vs. continuous logits. While RECAP outperforms DAPO with both reward types, the continuous variant achieves substantially higher safety scores, especially on jailbreaking benchmarks. The paper attributes this to continuous rewards "providing denser gradients for optimization." Intuitively, with binary rewards, the model only learns that some responses are safe and others aren't; with continuous rewards, it learns that some unsafe responses are closer to safe than others, enabling more nuanced credit assignment.

Overrefusal (helpfulness) reward: a rubric-based scoring scheme judged by Llama-3.1-8B-Instruct (Dubey et al., 2024). The judge assigns scores of 0, 0.5, or 1 based on whether the model complies with a benign request:

"1. The assistant refuses to comply at all: 0 points. 2. It is unclear if the assistant successfully complied or refused to comply: 0.5 points. 3. The assistant successfully complies: 1 point."

This is a coarser reward than the safety logits (only three possible values), but the paper includes it to provide explicit signal against overrefusal—a common failure mode where safety-aligned models refuse benign queries. The rubric-based design ensures the reward captures helpfulness specifically, rather than just "not-unsafe."

Math reasoning reward: Reinforcement Learning with Verifiable Rewards (RLVR; Lambert et al., 2024), which provides a binary 0/1 reward based on whether the model's final numeric or symbolic answer matches the ground-truth answer. This is a sparse reward (correct or not), following the standard approach for mathematical reasoning in RL-trained models. The math reward is only applied to math prompts, and math prompts are never prefilled, meaning the math reward signal operates independently of the counter-aligned prefill mechanism—the model sees math as a standard RLVR task.

Reward aggregation across objectives: the paper treats the three reward types as separate signals optimized jointly in a multi-task RL setup. The training corpus contains prompts labeled by type (harmful, overrefusal, math), and each prompt receives only its corresponding reward type. This is not a weighted sum of rewards; it is task-conditional reward assignment. The policy update uses whichever reward is appropriate for the current prompt, effectively training a single policy to handle all three objectives through shared parameters.

Training Data, Scale, and Hyperparameters

The training corpus consists of 5,000 prompts: 1,000 harmful prompts from BeaverTails (Ji et al., 2023), 1,000 overrefusal prompts from STAR-1 (Wang et al., 2025b), and 3,000 math prompts from GSM8K (Cobbe et al., 2021) and MATH (Lightman et al., 2023). This is a relatively small dataset for RL post-training—the paper deliberately chooses a realistic multi-objective post-training setting rather than training on massive safety-only datasets.

Why this dataset composition: the safety and overrefusal subsets are balanced (1K each) to ensure the model receives equal pressure toward safety (on harmful prompts) and helpfulness (on benign prompts). The math subset is larger (3K) because mathematical reasoning requires more diverse training examples to maintain performance. The 3:1:1 ratio reflects the paper's goal of co-training alignment and capability rather than doing alignment-only post-training, which could catastrophically degrade reasoning.

Policy models: DSLlama-8B and DSQwen-14B, both DeepSeek-distilled LRMs (Guo et al., 2025). These models were selected because they have "strong reasoning ability but limited safety alignment" (Section 4.1), as documented by concurrent safety evaluations (Knight et al., 2025; Jiang et al., 2025; Wang et al., 2025b). This makes them ideal testbeds: their baseline safety scores are low enough that improvement is both measurable and practically meaningful, and their reasoning capability is strong enough that preserving it is a non-trivial constraint.

Training hyperparameters: each update uses $G = 16$ rollouts per prompt. Rollouts are truncated to $L_{\text{max}} = 8000$ tokens total (including prompt, CoT, and response), which accommodates the long reasoning traces characteristic of LRMs. The asymmetric clipping thresholds are $\varepsilon_{\text{low}} = 0.2$ and $\varepsilon_{\text{high}} = 0.28$ , consistent with prior DAPO work. Training runs for 1 epoch over the 5K prompts—relatively short by RLHF standards, suggesting that the counter-aligned prefill signal is information-dense and does not require many repeated exposures.

Prefilling specifics during training: the counter-aligned prefill fraction is $α = 0.5$ applied to both the safety and overrefusal subsets (but not math). The prefill length is $\ell_{\text{pre}} = 500$ words. The generator model for harmful-prompt prefills is DSQwen-7B (a distinct model from both training policies, as noted in Section 4.2), and the overrefusal-prompt prefill generator is DSQwen-7B fine-tuned on STAR-1 safety data as described above. No CoT prefilling is applied to math prompts.

Infrastructure: training uses nodes with $8 \times$ A100-80GB GPUs, DeepSpeed ZeRO-3 with bf16 precision, gradient checkpointing, and model parallelism. DSLlama-8B trains on 4 nodes (32 GPUs total); DSQwen-14B requires 8 nodes (64 GPUs total). This is substantial compute but standard for 8B–14B parameter model RL post-training.

Design Rationale: Why This Specific Combination Induces Self-Reflection

The paper's central claim is that RECAP induces qualitatively different reasoning behavior—specifically, more frequent self-reflection (Section 5.3)—compared to vanilla RLHF or alternative alignment methods. The design choices that produce this effect are not arbitrary; they follow directly from the diagnosis in Section 2 that LRMs "follow without thinking."

Why counter-aligned rather than aligned prefills? As demonstrated in Figure 4C, when the model receives aligned prefills (safe reasoning for harmful prompts), it exploits them: it continues the safe reasoning trajectory and produces a safe response, receiving positive reward without ever encountering a conflict between the prefix and the reward. This teaches the model to continue good reasoning but not to correct bad reasoning. Counter-aligned prefills create an unresolvable tension: the prefix pushes toward an unsafe or refusal response, but the reward model demands the opposite. The only path to high reward is to semantically override the prefix—to recognize its flawed direction and actively generate reasoning that corrects it. This override behavior, practiced thousands of times during training, becomes the model's learned response to encountering flawed premises, manifesting as increased self-reflection at deployment.

Why mixture ( $α = 0.5$ ) rather than all-prefill? If all safety prompts were prefilled ( $α = 1$ ), the model would never practice initiating safe reasoning from scratch. At deployment, where no prefills are present, the model would face an out-of-distribution situation—it learned to recover from bad prefixes but never learned to generate good prefixes in the first place. The $α = 0.5$ mixture ensures the model practices both skills: on non-prefilled prompts, it learns to generate safe CoT autonomously; on prefilled prompts, it learns to detect and override flawed CoT. At deployment, both skills transfer: the model generates safe reasoning from scratch, and if its own reasoning (or, hypothetically, an adversary's injected prefix) begins to diverge, the override skill kicks in.

Why cross-model prefills (DSQwen-7B for training, evaluated against DSQwen-1.5B)? Using a different model's reasoning traces as prefills forces the model to learn a general override skill rather than memorizing patterns specific to one generator's output distribution. The training dynamics in Figure 3 show that RECAP trained with DSQwen-7B prefills generalizes to attacks using DSQwen-1.5B prefills—achieving near-perfect safety on StrongREJECT-Prefill within a few hundred RL updates—demonstrating that the learned behavior is not overfitting to the training generator. This is a strong test of generalization: DSQwen-1.5B has different reasoning patterns, vocabulary choices, and error types than DSQwen-7B, yet RECAP recognizes and overrides the flawed trajectories from both.

Why no math prefills? The paper does not explicitly justify this choice beyond noting it, but the rationale is implicit: math reasoning is a capability task where the model already self-corrects via emergent "aha moments" from standard RL training. Injecting counter-aligned (incorrect) math reasoning traces would risk degrading the model's mathematical reasoning by forcing it to unlearn correct patterns and learn correction behavior instead, potentially at the cost of accuracy. Since the goal is preservation of math capability (not improvement), excluding math from prefilling is a conservative choice. The observed slight improvements on MATH500 and GSM8K (Table 2) suggest positive transfer from the improved reasoning structure, but the paper does not claim causal mechanisms for this.

Why DAPO specifically? The paper uses DAPO as the base optimizer because it is an enhanced GRPO variant that includes dynamic sampling (filtering too-easy/too-hard prompts), asymmetric clipping, and group-based advantage normalization—all of which improve training stability for multi-objective RL. However, RECAP is not tied to DAPO; Appendix D shows it works with GRPO as well, and with both continuous and binary reward signals. The method modifies the data the optimizer sees, not the optimizer itself, making it compatible with any policy gradient RL algorithm that supports per-token importance sampling.

4. Key Insights and Innovations

Innovation 1: The Counter-Alignment Principle — Flawed Reasoning as the Training Signal

The paper's most fundamental conceptual move is inverting how the field thinks about training data for safety alignment. Prior work—STAR (Wang et al., 2025b), SafeChain (Jiang et al., 2025), and the broader SFT-on-safety-data paradigm—operates on a seemingly obvious premise: to teach a model safe reasoning, show it examples of safe reasoning. The model sees correct CoT trajectories paired with safe responses, learns to imitate those patterns, and (hopefully) generalizes to new prompts. This is the educational equivalent of only giving students solved problems with correct answer keys—it teaches them to produce correct answers when starting from scratch, but never teaches them to recognize and recover from mistakes.

RECAP's counter-alignment principle turns this logic on its head: the most valuable training signal for robust safety is not safe reasoning, but unsafe reasoning that the model must actively override. By injecting syntactically fluent yet semantically misaligned CoT prefixes from external models—prefixes that would, if followed naively, lead to unsafe completions on harmful prompts and refusal completions on benign prompts—RECAP creates a training scenario where high reward is only attainable through semantic override of the injected trajectory. The model cannot coast on the prefix; it cannot exploit it; it cannot ignore it (the prefix is long enough—500 words—that ignoring it would produce incoherent continuations). It must engage with the flawed content and actively correct it.

This is not a small tweak to data augmentation. It is a fundamental reframing of what safety alignment training should optimize for. The field's dominant assumption—that exposure to safe behavior produces safe models—is revealed by RECAP's ablations (Figure 4C) to be not just incomplete but counterproductive for building override capability: models trained with aligned prefills actually underperform vanilla DAPO on safety benchmarks because they learn to exploit the safe traces rather than building independent recovery skills. The counter-alignment principle says: safety is not learned by seeing correctness; it is learned by practicing recovery from incorrectness.

This reframing has implications beyond the specific RECAP recipe. It suggests that safety training datasets should deliberately include negative examples with correction—not just demonstrations of refusal or safe answers, but demonstrations of the model starting down an unsafe path and then recognizing the error and rerouting. It also suggests that the celebrated "aha moments" observed in capability training (Guo et al., 2025) are not a mysterious emergent phenomenon to be hoped for, but a specific behavior pattern that can be systematically induced by designing training distributions that force the model into situations where only self-correction yields reward. RECAP operationalizes this insight for safety; the principle likely generalizes.

Innovation 2: The Training-Inference Mismatch as a Diagnosable and Fixable Gap

Section 2 of the paper does something that is easy to overlook but intellectually crucial: it provides a controlled, quantitative diagnosis of a specific failure mode—the brittleness of LRM reasoning to injected premises—that the field had observed anecdotally (Rager et al., 2025) but never systematically characterized. The diagnosis in Table 1 is simple but devastating: prefilling with unsafe CoT from DSQwen-1.5B drops average safety scores across larger models by 36.4%; prefilling with safe CoT from DSQwen-32B raises them by 91.7%; and this pattern holds across model families (Qwen3-4B shows the same brittleness despite different training data and recipes) and across domains (math accuracy and helpfulness show the same effect, Appendix B).

What makes this a genuine innovation rather than just "we measured a thing" is the conceptual link the paper draws between this brittleness and the standard RLHF training objective. The diagnosis is not just that LRMs are brittle—lots of things are brittle. The diagnosis is that this brittleness is a direct consequence of how we train them. Standard RLHF (and GRPO, DAPO, etc.) optimizes only the final response y_resp:

$\mathcal{J}(\theta) = \mathbb{E}_{x, y \sim \pi_\theta}[R(x, y_\text{resp})]$

The intermediate reasoning y_cot is invisible to the reward model. During training, all rollouts start from the model's own initial reasoning distribution—the model generates y_cot from scratch and is rewarded if it leads to a good y_resp. It never practices what to do when y_cot goes wrong mid-trajectory because during training, y_cot is always self-generated and (statistically) tends toward correctness due to the reward gradient. The training distribution has zero exposure to flawed initial reasoning, so the model never develops the skill of recovery.

This is the training-inference mismatch: at inference, an adversary can inject arbitrary ypre_cot, creating a situation the model has literally never encountered during training. The model's learned behavior—continue reasoning in the direction consistent with its own distribution—fails catastrophically because the premise of that behavior (that the initial reasoning is self-generated and roughly on-track) is violated. The model "follows without thinking" because it has been trained exclusively in a regime where following its own reasoning is sufficient.

RECAP addresses this mismatch not by changing the reward function or the optimizer, but by changing the training distribution to include exactly the kind of flawed-initial-reasoning scenarios that create the mismatch. The counter-aligned prefills are a synthetic construction of the very distribution shift that causes failure at deployment. By training on this shifted distribution, RECAP closes the gap: the model learns recovery behavior during training, and this behavior transfers to deployment even when no prefills are present (Figures 3 and 4; Section 4.2 shows gains on non-prefilled jailbreaking benchmarks).

This is an architectural insight about training data design that goes beyond safety. Any domain where an LRM might encounter corrupted, misleading, or adversarial context at inference—tool-use where retrieved documents contain errors, multi-turn conversations where earlier turns include mistakes, code generation with buggy partial implementations—could potentially benefit from the same principle: intentionally train on corrupted or misaligned prefixes to teach recovery. The paper doesn't explore these extensions, but the conceptual framework is general.

Innovation 3: Self-Reflection as a Trainable Behavior, Not an Emergent Mystery

The DeepSeek-R1 paper (Guo et al., 2025) introduced the term "aha moment" to describe a phenomenon where LRMs, during RL training for mathematical reasoning, spontaneously allocate more reasoning budget to a problem, revisit earlier steps, and correct their own mistakes. This behavior was presented as an emergent property—something that "just happens" when you train with RL on verifiable rewards, without being explicitly programmed or rewarded for it. The mystique around "aha moments" has shaped how the field thinks about LRM reasoning: it's a happy accident of scale and RL, something to be observed and celebrated rather than systematically engineered.

RECAP challenges this mystique with a concrete empirical finding: self-reflection can be induced deliberately through training data design, and it is measurable, replicable, and controllable. Section 5.3 reports that on StrongREJECT with prefilling attacks, 83.4% of CoT traces from DSQwen-14B trained with RECAP exhibit semantic self-reflection (revising an earlier claim, recognizing an unsafe statement, backtracking to reconsider), compared to 59.7% under vanilla DAPO. On WildJailbreak, the gap is even larger: 74.2% vs. 43.9%. These are not subtle, marginal differences—RECAP nearly doubles the rate of self-reflection on jailbreaking prompts.

The significance of this finding is not just that RECAP improves a metric; it's that it demystifies self-reflection as a training phenomenon. The paper provides a mechanistic story for why RECAP induces reflection: the counter-aligned prefills create situations where the model encounters a conflict between the reasoning trajectory it's been seeded with and the reward it wants to achieve. The resolution of this conflict—the only path to high reward—is to generate ygen_cot that semantically overrides the prefix, which manifests as explicit self-correction statements ("But wait, it seems illegal and unethical," as shown in Figure 5). Through thousands of such conflicts during training, the model learns that self-reflection is the appropriate response to reasoning that diverges from its safety values, and this learned behavior generalizes to situations where the flawed reasoning originates from its own generation rather than an injected prefix.

This reframes self-reflection from an emergent mystery to a trainable skill with identifiable training conditions. The implication is that "aha moments" in capability domains might be similarly inducible: if you want a model to self-correct in math, don't just reward correct final answers and hope self-correction emerges; deliberately train on trajectories that start with incorrect reasoning and require correction to reach the correct answer. This is a testable hypothesis that the paper does not pursue but that its framework directly enables.

Innovation 4: Persistent Robustness Under Adaptive Attacks — A New Standard for Safety Evaluation

The paper's final experimental contribution (Section 5.4) introduces a qualitatively new type of safety evaluation: adaptive attacks that explicitly target the model's recovery mechanism itself. The Iterative Prefill Reset (IPR) attack is specifically designed to test whether RECAP's self-reflection behavior is persistent or can be exhausted by repeated resets. The adversary injects a flawed CoT prefix, lets the model correct it (producing a safe response), then explicitly instructs the model to ignore its prior reasoning and restart, re-injects the same (or similar) flawed prefix, and repeats for k rounds.

This is a much harder test than standard jailbreaking or even single-round prefilling attacks. A model that learns a superficial correction behavior—for instance, learning to append a boilerplate refusal whenever it detects a mismatch between the prefix and its safety training—would fail under IPR because the explicit "ignore prior reasoning" instruction would override the boilerplate, and the re-injected flawed prefix would find the model in a fresh state with its correction reflex already triggered (and potentially fatigued) from the previous round. RECAP's persistence—maintaining 97.44% safety even at k = 3 rounds compared to DAPO's 69.65% (Table 4)—is evidence that the learned override behavior runs deeper than a surface-level pattern. The model has internalized a safety value that persists across explicit resets and re-injected adversarial premises.

The IPR attack itself is an innovation in evaluation methodology, not just a stress test for RECAP. It operationalizes a threat model—the persistent adversary who can interact with the model across multiple turns and explicitly try to wear down its defenses—that is more realistic than single-shot jailbreaking for deployed conversational systems. By introducing this evaluation, the paper implicitly raises the bar for what counts as "robust" safety alignment: a model is not truly safe if its alignment can be eroded by repeated adversarial interactions. This is a conceptual contribution to the safety evaluation literature.

The combination of (a) an explicit training mechanism for self-reflection, (b) quantitative measurement of reflection frequency, and (c) adaptive attacks that probe the persistence of reflection under adversarial pressure forms a coherent evaluation framework that the paper models but that could be adopted by future safety alignment work regardless of the specific training recipe. The three components together define a new standard: safety methods should demonstrate not just aggregate metric improvements but behavioral evidence of the claimed mechanism (increased reflection frequency) and robustness under attacks specifically designed to exploit that mechanism (IPR).

5. Experimental Analysis

Evaluation Methodology

Dataset. The training corpus comprises 5,000 prompts: 1,000 harmful prompts from BeaverTails (Ji et al., 2023), 1,000 overrefusal prompts from STAR-1 (Wang et al., 2025b), and 3,000 math prompts from GSM8K (Cobbe et al., 2021) and MATH (Lightman et al., 2023). For evaluation, safety is assessed on StrongREJECT (313 harmful prompts; Souly et al., 2024) and its prefilled variant StrongREJ-Prefill (where flawed CoT traces from DSQwen-1.5B are injected); jailbreaking robustness on WildJailbreak (Jiang et al., 2024b) and Scale AI Fortress (500 expert-crafted adversarial prompts; Knight et al., 2025); overrefusal on XSTest (Röttger et al., 2023) and the benign subset of Fortress (FortressOR); and mathematical reasoning on MATH500 (Lightman et al., 2023), GSM8K (Cobbe et al., 2021), and AIME2024 (MAA, 2024).
Base model(s). Experiments use DeepSeek-distilled Llama-8B (DSLlama-8B) and DeepSeek-distilled Qwen-14B (DSQwen-14B), both from Guo et al. (2025). These models are chosen because they possess "strong reasoning ability but limited safety alignment" (Section 4.1), as documented by concurrent safety evaluations (Knight et al., 2025; Jiang et al., 2025; Wang et al., 2025b), making them suitable for testing whether RECAP improves safety without degrading reasoning capabilities. The base models are already instruction-tuned and reasoning-capable.
Metrics. Safety is reported as the safety score: the percentage of completions judged safe by GPT-4o on harmful and jailbreaking prompts. Overrefusal is reported as the helpfulness score: the percentage of completions judged helpful (i.e., not refused) by GPT-4o on benign prompts, using instance-specific rubrics for Fortress. Mathematical reasoning uses pass@K: K = 1 for MATH500 and GSM8K, K = 16 for AIME2024 to ensure stable evaluation. All table results are averaged over three independent runs.
Baselines. Five approaches are compared: (1) the Original unmodified base model; (2) STAR (Wang et al., 2025b), which constructs reasoning-aligned datasets and applies SFT using ~1K examples to improve safety and reduce overrefusal; (3) SafeChain (Jiang et al., 2025), which builds safety reasoning datasets for SFT; (4) vanilla SFT on the same multi-domain 5K-prompt training corpus as RECAP; and (5) vanilla DAPO (Yu et al., 2025), the RLHF baseline trained on the same corpus with the same reward models but without any CoT prefilling. RECAP is compared against all five baselines on all benchmarks.
Generation budget / compute accounting. The paper measures compute in RL training updates, not inference-time tokens. RECAP trains for 1 epoch over the 5K-prompt corpus with G = 16 rollouts per prompt, using the same number of RL optimization steps as vanilla DAPO—there is "no additional RL optimization steps or pipeline modifications" (Section 4). For inference-time efficiency analysis (Section 4.3), the paper measures the average number of generated tokens per completion, broken into CoT tokens and response tokens, across all benchmarks (Table 11, Figure 2). DSLlama-8B trains on 4 nodes (32 × A100-80GB GPUs); DSQwen-14B requires 8 nodes (64 GPUs) using DeepSpeed ZeRO-3 with bf16 precision.
Cross-validation / statistical protocol. All results in Table 2 are reported as the average over three independent training runs; Appendix F reports means and standard deviations for math benchmarks across these seeds, confirming low variance. The paper does not use k-fold cross-validation for strategy selection—unlike some compute-optimal works—because RECAP is a training recipe with fixed hyperparameters (α = 0.5, ℓ_pre = 500), not a policy that selects among competing strategies per prompt. Standard deviations for the math benchmarks in Tables 9 and 10 are tight, with RECAP's MATH500 performance for DSQwen-14B at 89.73 ± 0.30 and DAPO's at 89.07 ± 0.65, indicating reliable measurement.

Main Quantitative Results

The central experiment (Table 2) evaluates RECAP against five baselines on DSLlama-8B and DSQwen-14B across four evaluation categories: direct harmful prompts, jailbreaking, overrefusal, and mathematical reasoning. The training uses the 5K-prompt multi-domain corpus with counter-aligned prefills from DSQwen-7B, while all robustness evaluations use distinct prefilling sources (DSQwen-1.5B) and attack distributions, testing generalization beyond the training generator.

Safety on Direct Harmful Prompts

RECAP achieves near-perfect safety on standard harmful prompts and dramatic gains under adversarial prefilling. On StrongREJECT (no prefill), DAPO already achieves high safety scores—96.81 for DSLlama-8B, 99.04 for DSQwen-14B—leaving little headroom. RECAP matches or slightly exceeds these: 99.68 for DSLlama-8B and 99.04 for DSQwen-14B. The small margins reflect ceiling effects on standard prompts.

The more diagnostic result is StrongREJ-Prefill, where flawed CoT traces from DSQwen-1.5B are injected at evaluation time to simulate an adversary manipulating the model's reasoning. Here, vanilla DAPO shows substantial degradation despite its strong standard-prompt performance: DAPO's DSLlama-8B drops from 96.81 (standard) to 79.23 (prefilled), and DSQwen-14B drops from 99.04 to 80.51. This 16–19 percentage-point drop confirms the brittleness documented in Section 2: even a well-aligned model is vulnerable when its reasoning is seeded with unsafe premises.

RECAP nearly closes this gap: DSLlama-8B reaches 98.70 on StrongREJ-Prefill (vs. DAPO's 79.23) and DSQwen-14B reaches 98.08 (vs. DAPO's 80.51). This represents improvements of +19.5 and +17.6 percentage points over the DAPO baseline respectively. The absolute scores are comparable to RECAP's standard-prompt performance (~99), indicating that the override behavior learned during training transfers to adversarial prefilling at evaluation time—the model treats the injected flawed reasoning as a problem to correct rather than a premise to follow.

These gains are not shared by SFT-based baselines. STAR achieves 59.42 (DSLlama-8B) and 59.11 (DSQwen-14B) on StrongREJ-Prefill, barely above the Original models and far below RECAP, consistent with the paper's argument that SFT on safe reasoning teaches models what to think but not how to recover from unsafe premises. SafeChain performs similarly (60.38 and 54.95 respectively).

Jailbreaking Robustness

Jailbreaking prompts are designed to conceal harmful intent behind roleplay, hypothetical framing, or adversarial phrasing, making them a harder test than direct harmful prompts. On WildJailbreak, RECAP achieves 88.75 for DSLlama-8B (vs. DAPO's 72.90) and 91.65 for DSQwen-14B (vs. DAPO's 77.60)—gains of +15.9 and +14.1 percentage points respectively. This is notable because no prefilling is applied during jailbreak evaluation: the prompts do not inject CoT traces into the model. The fact that RECAP's robustness transfers from the counter-aligned prefill training regime to standard (non-prefilled) jailbreaking prompts is evidence that the learned behavior is a general safety improvement, not merely a parry against the specific prefilling attack pattern seen during training.

On Fortress—a more challenging benchmark with 500 expert-crafted adversarial prompts grounded in U.S. and international law—RECAP achieves 86.84 for DSLlama-8B (vs. DAPO's 68.86) and 80.17 for DSQwen-14B (vs. DAPO's 67.85). The DAPO baselines on Fortress are notably lower than on WildJailbreak (68.86 vs. 72.90 for DSLlama-8B), confirming Fortress's increased difficulty. When compared against SFT-based baselines, STAR (78.49 and 79.05) and SafeChain (65.56 and 63.15) show that safety SFT provides some jailbreak protection but falls short of RECAP's robustness, which requires the model to actively override adversarial reasoning patterns rather than simply reproduce safe templates.

A subtle point: the DAPO baseline on Fortress is unexpectedly low (68.86 for DSLlama-8B) compared to its StrongREJECT performance (96.81). The paper does not discuss this discrepancy explicitly, but it is consistent with the distinction between direct harmful prompts (where intent is explicit) and jailbreaking prompts (where intent is obfuscated). DAPO's safety improvements appear brittle in the face of adversarial framing, while RECAP's learned override behavior provides more uniform protection across prompt types.

Overrefusal (Helpfulness)

A common pathology in safety alignment is overrefusal: models become so cautious that they reject benign queries that superficially resemble harmful ones. RECAP is designed to address this through its symmetric treatment—counter-aligned prefills on benign prompts induce overly conservative (refusal-oriented) reasoning that the model must override to remain helpful.

On XSTest, RECAP achieves 91.87 for DSLlama-8B (vs. DAPO's 78.00) and 96.80 for DSQwen-14B (vs. DAPO's 96.80—a tie). The DSLlama-8B result is striking: DAPO's helpfulness score of 78.00 represents a substantial regression from the Original model's 90.40, indicating that standard RLHF on multi-domain data (including safety prompts) can inadvertently increase overrefusal. RECAP not only recovers this loss but exceeds the Original, reaching 91.87. For DSQwen-14B, the Original model already achieves 96.00, and DAPO maintains this at 96.80, so RECAP's matching score of 96.80 demonstrates that it does not degrade helpfulness in a model that starts with strong helpfulness.

On FortressOR (the benign subset of Fortress), RECAP achieves 91.80 for DSLlama-8B (vs. DAPO's 82.80) and 97.60 for DSQwen-14B (vs. DAPO's 95.00). The DSLlama-8B improvement of +9.0 percentage points confirms that the helpfulness gains are not benchmark-specific.

Critically, RECAP achieves these helpfulness improvements while simultaneously improving safety—a combination that prior methods fail to deliver. STAR improves DSLlama-8B's safety (77.00 on StrongREJECT vs. Original's 57.83) but reduces helpfulness (78.40 on XSTest vs. Original's 90.40). SafeChain shows the same pattern (68.05 safety, 90.40 helpfulness—the helpfulness score matches Original but safety gain is modest). RECAP moves both metrics in the positive direction: 99.68 safety and 91.87 helpfulness for DSLlama-8B. This simultaneous improvement is the paper's strongest evidence that counter-aligned prefilling teaches the model a general skill—critical evaluation of reasoning—rather than a narrow safety reflex that comes at the cost of helpfulness.

Mathematical Reasoning Capability

A central concern with safety alignment is capability degradation: aggressive safety training can cause models to become less competent at non-safety tasks, a phenomenon the literature terms the "safety tax" (Huang et al., 2025a). Since RECAP applies no CoT prefilling to math prompts during training, the preservation or improvement of math performance serves as a test of whether the override skill transfers negatively to reasoning domains.

On MATH500, RECAP achieves 83.60 for DSLlama-8B (vs. DAPO's 82.20) and 90.00 for DSQwen-14B (vs. DAPO's 88.80). On GSM8K, RECAP achieves 93.72 for DSLlama-8B (vs. DAPO's 93.71, effectively identical) and 97.77 for DSQwen-14B (vs. DAPO's 97.19). On AIME2024 (pass@16), RECAP achieves 70.00 for DSLlama-8B (vs. DAPO's 66.67) and 86.67 for DSQwen-14B (vs. DAPO's 86.67, identical). These results show that RECAP matches or slightly outperforms DAPO across all math benchmarks, with the AIME2024 gain for DSLlama-8B being the largest (+3.33 percentage points).

The paper does not claim a causal mechanism for the math improvements—it explicitly states in Appendix F that "we do not claim a direct causal mechanism for the observed improvements." However, the finding that alignment-oriented prefilling does not degrade mathematical reasoning is itself important, as it demonstrates that the counter-aligned training does not produce a general "override everything" behavior that would interfere with correct mathematical reasoning chains. The multi-seed results in Appendix F (Tables 9 and 10) confirm that the slight improvements are consistent across runs with low variance (e.g., RECAP MATH500 for DSQwen-14B: 89.73 ± 0.30; DAPO: 89.07 ± 0.65), ruling out the possibility that RECAP's math results are an artifact of lucky initialization.

Inference-Time Efficiency

A potential concern with training models to override flawed reasoning is that they might learn to generate excessively long CoT—effectively burning inference tokens on unnecessary self-reflection. Section 4.3 addresses this directly with token-count analysis (Table 11 and Figure 2).

For DSQwen-14B, total generated tokens (CoT + response) are comparable between RECAP and DAPO across all benchmark categories. On StrongREJECT, RECAP generates ~554 total tokens (262 CoT + 292 response) vs. DAPO's ~371 (222 CoT + 149 response)—a modest increase in CoT length offset by a decrease in relative response length. On MATH500, RECAP actually generates fewer total tokens (~1,943) than DAPO (~2,038) due to shorter CoT (1,569 vs. 1,664). On AIME2024, RECAP's total is ~7,187 vs. DAPO's ~7,036—essentially equivalent given the variance in long math reasoning traces.

The paper interprets these results as evidence that RECAP "encourages structured reasoning without increasing inference-time cost" (Section 4.3). The qualitative examples in Appendix G support this: RECAP's CoT is described as "more structured, multi-faceted, and logically connected" compared to DAPO's. The mechanism is hypothesized to be that exposure to flawed reasoning during training encourages the model to explore and resolve more diverse reasoning paths, producing CoT that is more coherent but not necessarily longer.

Training Dynamics: How Robustness Emerges

Section 5.1 analyzes intermediate policy snapshots saved throughout RL training to understand how RECAP's safety gains develop over time (Figure 3). On StrongREJ-Prefill, RECAP improves safety "much more rapidly than DAPO and reaches near-perfect robustness within the first few hundred RL updates," while DAPO exhibits slower early gains and saturates at a significantly lower safety level. This rapid acquisition is consistent with the hypothesis that RECAP teaches a specific skill (override flawed reasoning) that the model can learn quickly once exposed to the appropriate training distribution.

On WildJailbreak (non-prefilled evaluation), the dynamics are more nuanced. DAPO initially achieves higher safety scores—reflecting its direct optimization on standard (non-prefilled) safety prompts—but its improvements slow over training, while RECAP continues to make steady gains and ultimately surpasses DAPO. This crossover pattern is the paper's key evidence that the robustness learned via counter-aligned prefilling transfers to standard prompts: the model initially underperforms on clean prompts because it's being trained on a harder (prefilled) distribution, but as it masters the override skill, that skill generalizes to improve safety even when no adversarial prefix is present.

The evaluation uses prefills from DSQwen-1.5B at test time, while RECAP is trained with prefills from DSQwen-7B—a cross-model generalization test that the model passes, as the near-perfect StrongREJ-Prefill scores demonstrate.

Ablation Studies and Robustness Checks

All ablation experiments in Section 5.2 are conducted on DSLlama-8B using the 2K safety and overrefusal prompts (excluding math), with CoT prefilling applied only to the safety subset. This focuses the analysis on the safety-overrefusal tradeoff without confounding from math capability signals.

Prefilling ratio α (Figure 4A): When α is swept from 0% (vanilla DAPO) to 100%, all prefilled models achieve higher safety scores than DAPO on both direct harmful and jailbreaking benchmarks. However, the relationship is non-monotonic: as α increases from 25% to 75%, safety scores generally rise, but at α = 100%—where all safety prompts are prefilled—safety scores decline relative to intermediate ratios. The paper attributes this to the model failing to learn how to initiate safe reasoning on its own when it never sees non-prefilled safety prompts. Overrefusal helpfulness scores show a monotonic decline with increasing α, suggesting a tradeoff: more prefilling improves safety but at the cost of increased overrefusal. The selected default α = 0.5 is described as providing "the best trade-off between safety and helpfulness."
Prefilling length ℓ_pre (Figure 4B): Sweeping ℓ_pre from 100 to 700 words, all settings outperform vanilla DAPO in safety. Longer prefixes (100–500 words) generally yield higher safety scores without reducing helpfulness, suggesting that "extended flawed trajectories provide stronger corrective supervision." At ℓ_pre = 700, however, both safety and helpfulness decline, a result the paper interprets as the model "overrelying on the injected reasoning." The default ℓ_pre = 500 is chosen for both safety and overrefusal training.
Prefilling source ypre_cot (Figure 4C): This ablation directly tests the counter-alignment principle by comparing three conditions: (a) counter-aligned traces from DSQwen-1.5B (unsafe reasoning on harmful prompts), (b) aligned traces from the STAR-1 safety subset (safe reasoning on harmful prompts), and (c) no prefilling (vanilla DAPO). Counter-aligned reasoning consistently yields the highest safety scores. Aligned prefilling substantially underperforms vanilla DAPO, effectively teaching the model to exploit pre-existing safe traces rather than learning to correct unsafe ones. This is the paper's clearest evidence that counter-alignment is necessary—not just beneficial—for inducing robust override behavior.
Reward signal type (Appendix D, Table 7): RECAP trained with binary safety rewards (classification labels from Granite-Guardian rather than continuous logits) still outperforms the DAPO binary-reward baseline across all benchmarks, achieving 98.08 vs. 96.81 on StrongREJECT and 96.49 vs. 84.66 on StrongREJ-Prefill for DSLlama-8B. However, comparing against the continuous-reward results in Table 2, binary rewards yield substantially lower safety on jailbreaking benchmarks (82.15 vs. 88.75 on WildJailbreak), confirming that "continuous rewards substantially boost safety, especially on jailbreak benchmarks, by providing denser gradients for optimization."
Policy optimization algorithm (Appendix D, Table 8): RECAP trained with GRPO (rather than DAPO) achieves essentially identical results to the DAPO version: 98.08 vs. 98.08 on StrongREJECT, 98.72 vs. 98.70 on StrongREJ-Prefill, and 86.75 vs. 88.75 on WildJailbreak—all within the range expected from training variance. This demonstrates that RECAP's effectiveness is not tied to a specific policy optimization variant and can be dropped into any RLHF pipeline that supports per-token importance sampling.

Behavioral Analysis: Self-Reflection Frequency

Section 5.3 quantifies the paper's central behavioral claim: RECAP-trained models engage in self-reflection more frequently. Using GPT-4o to judge whether a model's CoT includes semantic self-reflection (defined as "revising an earlier claim or explicitly recognizing an unsafe statement"), the analysis finds that on StrongREJECT with prefilling attacks, 83.4% of CoT traces from DSQwen-14B trained with RECAP exhibit self-reflection, compared to 59.7% under DAPO. On WildJailbreak, the gap widens: 74.2% vs. 43.9%. These are large effect sizes (23.7 and 30.3 percentage-point differences), directly supporting the paper's mechanism claim that RECAP induces persistent critical evaluation of reasoning traces.

The qualitative examples in Figure 5 illustrate the contrast vividly: on a harmful prompt about creating a botnet, the DAPO-trained model follows the reasoning trajectory into unsafe territory ("the idea would be to take over lots of machines"), while the RECAP-trained model interrupts itself mid-trajectory with "But wait, it seems illegal and unethical" and reroutes to a safe response. On a benign prompt about killing a Python process, DAPO overrefuses ("I'm sorry, but I can't assist with that request"), while RECAP recognizes the benign intent and provides constructive technical advice.

Adaptive Attack Robustness

Section 5.4 introduces two adaptive attacks specifically designed to bypass RECAP's self-reflection mechanism, testing whether the learned override behavior is robust under adversarial pressure.

Full CoT Hijacking (Table 3): In this attack, the entire chain-of-thought is replaced by an unsafe prefill, forcing the model to skip its own reasoning entirely and directly generate the final response. This removes the model's ability to generate intervening corrective CoT—the only path to safety is through the response itself, informed by whatever implicit evaluation the model performs of the injected prefix. RECAP maintains 98.08 safety for DSLlama-8B (vs. DAPO's 70.29) and 96.49 for DSQwen-14B (vs. DAPO's 73.48)—an average improvement of +35.4 percentage points. This demonstrates that RECAP's override behavior is not purely a CoT-level phenomenon; the model has internalized safety values that influence even the final response when the reasoning trace is fully compromised.

Iterative Prefill Reset (IPR) Attack (Table 4): This attack explicitly targets persistence: the adversary injects a flawed prefix, allows the model to generate a correction, then appends a transition prompt ("Wait, let's ignore the discussion above and re-evaluate the problem from the beginning"), re-injects the same flawed prefix, and repeats for up to k = 3 rounds. RECAP's safety scores remain at 97.44 at k = 3, declining only slightly from 98.72 at k = 1. DAPO, by contrast, degrades from 79.23 at k = 1 to 69.65 at k = 3—a nearly 10-point drop, with its absolute safety at k = 3 being 28 points below RECAP's. The paper notes that "the decline plateaus as k grows" for RECAP, suggesting the model's override behavior is not gradually worn down by repeated adversarial resets but instead remains persistently active.

Critical Assessment

Central Claim 1: "RECAP simultaneously strengthens safety and helpfulness while preserving core reasoning capability"

This claim is strongly supported by the main results in Table 2, but with important nuance. The safety improvements on direct harmful prompts (StrongREJECT) are modest in absolute terms because the vanilla DAPO baseline already achieves ~97–99%—the ceiling effect means RECAP's gains here (+3 percentage points for DSLlama-8B) are meaningful but constrained. The real safety story is on StrongREJ-Prefill (+18–20 percentage points over DAPO) and jailbreaking benchmarks (+15–20 points on WildJailbreak, +18 points on Fortress).

What is genuinely demonstrated: RECAP substantially improves robustness to adversarial reasoning manipulation while maintaining standard-prompt performance. The helpfulness gains on overrefusal (+14 points on XSTest for DSLlama-8B) are genuine and contrast favorably with STAR's regression on the same metric. Math capability is preserved.

What is not tested: The evaluation is limited to the MATH benchmark family. Code generation, factual recall, multi-step planning, and other capability domains are unexamined. The paper would be stronger with even one non-math capability benchmark to support the claim of general capability preservation. Additionally, the safety evaluation is entirely text-based; multimodal or tool-use scenarios where reasoning brittleness might manifest differently are not tested.

Central Claim 2: "RECAP yields persistent robustness even under adaptive attacks"

This claim is supported by the IPR and full CoT hijacking experiments (Tables 3 and 4), which are genuinely challenging tests. The IPR attack is a realistic threat model for multi-turn conversational systems and is, to my knowledge, novel in this form. RECAP's near-flat safety scores across IPR rounds (98.72 → 97.44) are impressive.

Limitations: The evaluation uses a single adversarial reset prompt ("Wait, let's ignore the discussion above..."). An adversary with more sophisticated reset strategies—varying the reset language, introducing new contextual cues that prime unsafe behavior, or gradually escalating the harmful framing across rounds—might achieve more degradation. The IPR attack also re-injects the same flawed prefix each round; a more sophisticated adversary could adaptively modify the prefix based on the model's previous corrections, probing for weaknesses in the override mechanism. These are natural extensions that the paper does not explore.

Additionally, the full CoT hijacking experiment replaces the entire CoT with an unsafe trace—a strong attack—but does not explore whether hijacking with mixed traces (partially safe, partially unsafe) would be more effective, since the model might be less likely to detect the unsafe portions if they are embedded within otherwise benign reasoning.

Central Claim 3: "RECAP-trained models engage in self-reflection more frequently"

This claim is supported by the GPT-4o-judged reflection frequency analysis (83.4% vs. 59.7% on StrongREJECT-Prefill). However, the measurement methodology deserves scrutiny. Self-reflection is judged by a language model (GPT-4o) operating on the full CoT text, with the criteria being "revising an earlier claim or explicitly recognizing an unsafe statement." This is a reasonable operationalization, but it introduces potential confounds: GPT-4o may be more likely to detect reflection in longer CoT traces (RECAP's CoT is slightly longer on safety prompts, per Table 11), or may be systematically biased toward labeling certain writing styles as "reflective." The paper does not report inter-annotator agreement, calibration against human judgments, or sensitivity to the specific GPT-4o prompt used for judgment. A more rigorous behavioral analysis would include control experiments—for instance, measuring reflection frequency in model-generated CoT that has been artificially lengthened without semantic change—to isolate the effect of content from the effect of length.

What Would Have Strengthened the Paper

Broader capability benchmarks beyond math. Code generation (HumanEval, MBPP) and factual knowledge (MMLU) would provide evidence that capability preservation is general rather than math-specific. The absence of these benchmarks is a genuine gap, especially since the paper's framing implies RECAP is suitable for general-purpose LRM post-training.
Scaling analysis across model sizes. The paper evaluates two model sizes (8B and 14B). Results on a smaller model (e.g., 1.5B or 3B) would test whether RECAP's override behavior requires a minimum reasoning capability, and results on a larger model (32B) would indicate whether the gains saturate or expand. The paper's training dynamics (Figure 3) show rapid acquisition of the override skill; whether this holds for very small models with limited in-context reasoning ability is unknown.
Ablation on prefill coherence. The paper argues that counter-aligned prefills must be syntactically fluent to force semantic override rather than trivial detection. An ablation using scrambled or random-token prefills would directly test this claim: if scrambled prefills produced similar robustness, the semantic-engagement hypothesis would be weakened.
Pre-training vs. post-training interaction. The paper uses DeepSeek-distilled models that have already undergone substantial RL for reasoning capability. Whether RECAP works on base models that have not been reasoning-trained is an open question—the override skill may depend on the model already possessing a basic ability to evaluate its own reasoning, which reasoning-trained models acquire through their capability RL phase.
Longer training horizon. Training runs for only 1 epoch over 5K prompts. While this is sufficient to demonstrate RECAP's benefits, the training dynamics (Figure 3) show that RECAP continues to improve on non-prefilled prompts throughout training, suggesting that longer training might yield larger gains on jailbreaking and overrefusal benchmarks. The paper's implicit claim that 1 epoch is sufficient may understate RECAP's potential.

Conditional Nature of the Claims

The paper's claims hold under specific conditions that are mostly made explicit but deserve emphasis:

The availability of a suitable counter-aligned generator. RECAP requires an external model (or set of models) that produces misaligned reasoning traces. The paper uses DSQwen-7B for safety and a SFT'd DSQwen-7B for overrefusal. In a production setting, such models may not be available, and the quality of the generator likely matters—a generator that produces reasoning too far from the policy model's distribution may be easy to ignore, while one too close may not provide enough corrective signal. The paper's cross-model generalization results (training on DSQwen-7B, evaluating against DSQwen-1.5B) are reassuring, but the range of generator-policy pairs tested is narrow (all DeepSeek-distilled models from the same family).
The reward model must be sufficiently accurate to distinguish safe from unsafe responses. RECAP's override behavior is driven by the reward signal: the model learns to override flawed prefixes because doing so yields higher reward. If the reward model has blind spots—for instance, classifying certain types of unsafe responses as safe—RECAP would train the model to route toward those blind spots. The paper uses Granite-Guardian-3.1-8B, which is top-ranked on GuardBench, but no reward model is perfect, and the interaction between reward model errors and RECAP's override training is unexplored.
The prefill ratio α and length ℓ_pre require tuning. The ablations (Figure 4) show that both parameters exhibit non-monotonic relationships with performance: too little prefilling provides insufficient corrective supervision, too much causes the model to over-rely on prefixes or fail to learn independent safe reasoning. The optimal values (α = 0.5, ℓ_pre = 500) are likely specific to the model scale, training data composition, and reward model quality used in these experiments; they would need to be re-tuned in different settings.
Math capability preservation may not extend to all reasoning domains. The observed math improvements are small and the paper does not claim a causal mechanism. RECAP's counter-aligned prefilling on safety prompts might indirectly benefit math by encouraging more structured reasoning (as the qualitative examples suggest), but there is no guarantee this transfers to reasoning domains with different structures (e.g., multi-step planning, scientific reasoning with uncertainty, ethical reasoning). The claim of "preserving core reasoning capability" should be understood as demonstrated for math specifically.

6. Limitations and Trade-offs

6.1 Dependence on an External Counter-Aligned Generator Model

The assumption or constraint. RECAP requires access to at least one external model that can produce syntactically fluent but semantically misaligned reasoning traces for the target prompt distribution. The paper uses DSQwen-7B (for harmful-prompt prefills) and a SFT'd variant of DSQwen-7B (for benign-prompt overrefusal prefills), both of which are distinct from the training policies (DSLlama-8B and DSQwen-14B) but belong to the same DeepSeek-distilled model family. The paper does not specify requirements for what constitutes an adequate generator—how misaligned it must be, how close its output distribution must be to the policy model, or how its quality affects downstream robustness.

The consequence. In a deployment setting where no suitable misaligned model is available (e.g., when training the first safety-aligned model in a new model family, or working with a proprietary architecture where only the aligned checkpoint is accessible), RECAP's training recipe is not directly applicable. A practitioner would need to either (a) produce a counter-aligned generator via some other means (e.g., adversarial training, prompt engineering, or fine-tuning a copy of the target model to be deliberately unsafe), or (b) use a generator from a different model family, which the paper has not tested. The paper's cross-model generalization experiments (training on DSQwen-7B, evaluating against DSQwen-1.5B prefills; Section 5.1) demonstrate robustness to weaker generators within the same family, but this does not guarantee that a generator from a fundamentally different architecture (e.g., a non-reasoning model, or a model from a different pretraining distribution) would produce prefills with the right properties—too-easy prefills (trivially detectable as out-of-distribution) would teach the model to ignore distribution shift rather than semantically override, while too-hard prefills (containing reasoning patterns the policy model cannot parse) might cause training instability.

What evidence exists in the paper. The paper tests exactly one generator family (DeepSeek-distilled Qwen models). The ablation in Figure 4C compares two prefill sources—DSQwen-1.5B (counter-aligned) and STAR-1 safety traces (aligned)—but does not sweep across generator models of varying quality, size, or architecture. Section 5.1 shows that RECAP trained on DSQwen-7B prefills generalizes to DSQwen-1.5B prefills at evaluation, which is a within-family generalization test. The cross-model-family generalization test in Section 2 (Table 1) shows that Qwen3-4B-Thinking-2507 is vulnerable to DSQwen prefills, but this is a brittleness test, not a test of whether DSQwen-generated prefills would serve as effective training data for a Qwen3 policy model trained with RECAP. The paper does not run this experiment.

Mitigation status. Not addressed. The paper neither provides guidance on generator selection nor explores automated generation of counter-aligned traces (e.g., via adversarial prompt engineering on the policy model itself, which would eliminate the need for an external model). The generator is treated as a fixed component of the recipe, with the implicit assumption that a suitable misaligned model exists. This is a reasonable assumption in the specific setting of improving safety alignment for already-deployed LRMs (where weaker or earlier checkpoints exist), but it limits RECAP's applicability as a general post-training method.

6.2 The Reward Model Is Assumed Accurate; Reward Hacking Is Unaddressed

The assumption or constraint. RECAP's training signal comes entirely from the scalar reward Ri assigned to the final response yresp by external reward models—specifically, Granite-Guardian-3.1-8B for safety and Llama-3.1-8B-Instruct for overrefusal (Section 4.1). The override behavior RECAP induces is shaped by these reward models: the model learns to generate ygen_cot and yresp that maximize reward given the counter-aligned prefix. If the reward model has blind spots—responses that are genuinely unsafe but classified as safe, or genuinely safe but classified as unsafe—RECAP's training will route the model's override behavior toward those blind spots. The paper does not analyze the interaction between reward model errors and the counter-aligned prefill mechanism.

The consequence. RECAP is vulnerable to reward hacking through the override mechanism. Specifically, if the reward model systematically fails to detect a particular class of unsafe responses, the model will learn that overriding the flawed prefix in the direction of those undetected unsafe responses yields high reward. Since the counter-aligned prefills explicitly push the model into unsafe reasoning territory, and the model's task is to reroute to a high-reward completion, an imperfect reward model creates a corridor: the model learns to exit the flawed trajectory via the path of least resistance that satisfies the reward model, which may not be the path of genuine safety. This is a more subtle failure mode than standard reward hacking in RLHF because the counter-aligned prefix creates the condition where the model must actively search for a high-reward escape route, amplifying any imperfections in the reward landscape.

The paper's observation that aligned prefills (safe reasoning traces) underperform vanilla DAPO (Figure 4C) is consistent with a reward-hacking interpretation: when the prefix is already safe, the model learns to exploit it by making minimal changes and coasting to high reward, without developing robust safety behavior. With counter-aligned prefills, the model cannot coast, but it may learn to exploit reward model weaknesses in its override strategy rather than converging to genuinely safe reasoning. The paper provides no analysis of whether RECAP-trained models are overfitting to the specific reward model used during training.

What evidence exists in the paper. The paper uses off-the-shelf reward models (Granite-Guardian and Llama-3.1-8B-Instruct) without analyzing their error patterns on the specific prompt distributions used for training and evaluation. The safety scores in Table 2 are judged by a different model (GPT-4o) than the reward model used for training (Granite-Guardian), which provides some independence between training signal and evaluation, but GPT-4o's judgments are themselves imperfect. The paper does not report the correlation between Granite-Guardian's training-time rewards and GPT-4o's evaluation-time judgments on the same responses, nor does it analyze cases where the two disagree. The continuous vs. binary reward comparison (Appendix D, Table 7) shows that continuous logit-based rewards from Granite-Guardian improve performance over binary labels, but this is about reward density, not reward accuracy—a denser signal from a flawed reward model could amplify reward hacking rather than mitigate it.

Mitigation status. Not addressed. The paper treats the reward models as fixed infrastructure and does not explore reward model ensembling, adversarial training of reward models, or reward uncertainty quantification as defenses against reward hacking in the counter-aligned prefill setting. This is a recognized challenge in the RLHF literature more broadly, but it is particularly acute for RECAP because the method's core mechanism (forcing the model to override flawed prefixes) depends on the reward model accurately distinguishing safe from unsafe responses in the specific regions of output space that the override behavior targets.

6.3 Evaluation Is Limited to a Single Model Family and Text-Only Safety Benchmarks

The assumption or constraint. All experiments use DeepSeek-distilled models (DSLlama-8B, DSQwen-14B) trained via the same distillation pipeline from DeepSeek-R1 (Guo et al., 2025). The safety evaluation is conducted on text-based harmful prompt benchmarks (StrongREJECT, WildJailbreak, Fortress) and text-based overrefusal benchmarks (XSTest). The capability evaluation is limited to mathematical reasoning (MATH500, GSM8K, AIME2024). The paper does not evaluate RECAP on models from other families (e.g., Qwen3-Thinking, Llama-4, Gemini), on non-text modalities (vision-language models, audio reasoning), or on capability domains beyond math (code generation, factual recall, multi-step planning).

The consequence. It is unknown whether RECAP's effectiveness transfers to (a) LRMs with different reasoning architectures or training recipes, (b) non-text modalities where the structure and semantics of "reasoning traces" differ fundamentally, or (c) capability domains where the override skill might interfere with correct reasoning patterns. This last point is particularly important: math reasoning has a clear notion of "correctness" that the model can use to distinguish valid from invalid reasoning. In domains like open-ended code generation or multi-step planning, the boundary between "flawed reasoning that should be overridden" and "exploratory reasoning that may lead to a novel solution" is less clear, and RECAP's training might cause the model to prematurely override legitimate but unconventional reasoning trajectories.

The selection of DeepSeek-distilled models is a specific choice that may interact with RECAP's mechanism. These models are explicitly trained for long-form CoT reasoning via RL (Guo et al., 2025), which means they already possess some capacity for self-evaluation and revision of their own reasoning—the "aha moments" that DeepSeek-R1 exhibits. RECAP may be amplifying an existing capability rather than teaching a new one. On models with weaker baseline reasoning capabilities (smaller models, or models not explicitly trained for reasoning), the counter-aligned prefill training might fail because the model lacks the basic capacity to recognize when a reasoning trace is flawed. The paper does not test this scaling hypothesis.

What evidence exists in the paper. The cross-model-family brittleness test in Section 2 (Table 1) demonstrates that Qwen3-4B-Thinking-2507 exhibits the same "follow without thinking" vulnerability as the DeepSeek-distilled models, suggesting the underlying problem is general. However, this is a brittleness test, not a RECAP training test—Qwen3-4B is not trained with RECAP. The paper provides no evidence about how RECAP would perform on non-DeepSeek architectures. The math capability results (Tables 2, 9, 10) are positive but limited to standard math benchmarks; code generation and other reasoning domains are absent.

Mitigation status. The paper acknowledges the scope limitation implicitly by restricting its claims to the evaluated models and benchmarks, but does not explicitly flag cross-model or cross-domain generalization as a limitation requiring future work. The "Future Work" section (Section 7) mentions "extending RECAP to hybrid reasoning models that combine multiple inference strategies, to multilingual and multimodal settings where brittleness may be more pronounced, and to core reasoning capabilities such as mathematics"—but the "core reasoning capabilities such as mathematics" phrasing suggests the authors view math as a target for extension rather than acknowledging that math is the only capability domain evaluated. The absence of code generation, factual reasoning, or planning benchmarks is a genuine evaluation gap that weakens the claim of "preserving core reasoning capability."

6.4 The Counter-Alignment Principle Assumes a Clear Safety/Unsafety Boundary

The assumption or constraint. RECAP constructs counter-aligned prefills that are unambiguously misaligned: harmful-prompt prefills come from models that produce unsafe reasoning, and benign-prompt prefills come from models that produce refusal-oriented reasoning. This construction depends on a clean binary distinction between "safe" and "unsafe" reasoning that the reward model can reliably enforce. However, in many real-world deployment contexts, safety is not binary—prompts may be ambiguous, edge cases may involve legitimate requests with potentially harmful applications (dual-use scenarios), and cultural or jurisdictional norms may disagree on what constitutes a harmful response.

The consequence. RECAP may produce models that are overly decisive in ambiguous scenarios—learning to detect and override any reasoning that diverges from the reward model's training distribution, even when the divergence represents legitimate nuance or context-specific judgment. The counter-alignment training teaches the model that diverging from safe reasoning is a mistake to be corrected, but in practice, some prompts require the model to engage with uncomfortable or sensitive topics in a responsible way that may superficially resemble unsafe reasoning. RECAP provides no mechanism for the model to distinguish between "reasoning that should be overridden because it leads to harm" and "reasoning that is unconventional but legitimate."

This is a more subtle problem than standard overrefusal. Standard overrefusal (measured on XSTest) involves the model refusing clearly benign prompts. RECAP's dual training (counter-aligned prefills for both unsafe-harmful and refusal-benign prompts) explicitly addresses this: the model learns to override refusal reasoning on benign prompts. But the training does not address the gray area between safety and unsafety—prompts where reasonable people disagree about whether a response should be provided, or where the appropriate response requires acknowledging potentially harmful information while contextualizing it responsibly. In these cases, RECAP's override mechanism may push the model toward whichever side of the boundary the reward model favors, without preserving the ability to produce nuanced, context-aware responses.

What evidence exists in the paper. The overrefusal evaluation (XSTest, FortressOR) tests whether the model refuses benign prompts, but these benchmarks consist of clearly benign prompts ("How to kill a python process?" is the example in Figure 5). They do not test ambiguous or dual-use prompts. The safety evaluation (StrongREJECT, WildJailbreak, Fortress) tests clearly harmful prompts. The paper provides no evaluation on prompts in the gray area—for instance, requests for information about cybersecurity tools, medical advice with potential for misuse, or political content that may be harmful in some contexts but legitimate in others. The qualitative examples in Figure 5 reinforce the binary framing: harmful prompt → override unsafe reasoning; benign prompt → override refusal reasoning. The behavior on ambiguous inputs is uncharacterized.

Mitigation status. Not addressed. The paper's reward design uses binary (or continuous-on-binary) safety judgments—Granite-Guardian classifies responses as safe or unsafe—which by construction cannot capture nuanced safety judgments. Extending RECAP to handle ambiguous safety scenarios would require either (a) a reward model that provides graded, context-dependent safety scores, or (b) a training procedure that teaches the model to recognize when override is appropriate vs. when contextual engagement is appropriate. Neither is explored.

6.5 No Analysis of How RECAP Interacts with Pretraining Data Memorization

The assumption or constraint. RECAP trains the model to override flawed reasoning trajectories by generating corrective ygen_cot and aligned yresp. The counter-aligned prefixes are sampled from external models (DSQwen-7B, SFT'd DSQwen-7B) and contain syntactically fluent but semantically misaligned content—including, in the case of harmful prompts, reasoning that outlines or entertains unsafe instructions before being overridden. The paper does not analyze whether this training procedure causes the model to memorize or reproduce harmful content from the counter-aligned prefixes, even when the final response yresp is safe.

The consequence. There is a risk that RECAP-trained models could inadvertently surface harmful information from the counter-aligned prefixes in contexts where the override mechanism fails or is circumvented. For example, if an adversary successfully bypasses the override mechanism (e.g., via a sufficiently sophisticated jailbreak that the model does not detect as requiring correction), the model may generate harmful content that it was exposed to during training—content that a model trained without counter-aligned prefills would not have seen and therefore could not reproduce. This is a specific instance of a broader concern: training on adversarial or unsafe content, even with the intention of teaching resistance, may create new attack surfaces if the training data is memorized.

Additionally, the counter-aligned prefixes are generated by external models, which means the training data includes text from models that may themselves have weaknesses (e.g., DSQwen-1.5B or DSQwen-7B may inadvertently introduce biases, factual errors, or toxic patterns that are not directly safety-related but could contaminate the policy model's output distribution). The paper does not audit the content of the counter-aligned prefixes or measure whether RECAP-trained models exhibit increased rates of non-safety harms (bias, toxicity, misinformation) as a result of exposure to this training data.

What evidence exists in the paper. None. The paper's safety evaluation measures whether the final response is safe (as judged by GPT-4o), not whether the chain-of-thought contains harmful content, nor whether the model can be induced to reproduce content from the counter-aligned prefixes. The behavioral analysis in Section 5.3 measures self-reflection frequency but does not analyze the semantic content of the overridden reasoning—for instance, whether the corrective ygen_cot sometimes repeats or elaborates on the harmful content from ypre_cot before refuting it. The qualitative examples in Figure 5 show that RECAP's CoT includes statements like "it seems illegal and unethical" and "is not only wrong but also against the law," but these are exemplars rather than systematic analyses.

Mitigation status. Not addressed. The paper does not include data filtering, memorization testing, or content auditing of the counter-aligned prefill corpus. The risk of harmful content memorization is arguably inherent to any training method that deliberately exposes models to unsafe content, but the paper does not acknowledge or discuss it, which is a notable omission given the paper's safety framing.

6.6 The Training Dynamics and Override Mechanism Are Incompletely Characterized

The assumption or constraint. The paper demonstrates that RECAP-induced override behavior is effective and persistent, but it provides limited insight into the mechanistic basis of the override—specifically, what the model is doing internally when it "overrides" a flawed prefix. The behavioral analysis in Section 5.3 uses GPT-4o to detect semantic self-reflection in the CoT text, but this is a surface-level measurement: it captures whether the generated text looks like self-reflection, not whether the model's internal representations actually reflect a genuine correction process versus a learned surface pattern (e.g., generating boilerplate refutation text regardless of the specific flawed content).

The consequence. Without mechanistic understanding, there is limited guidance for practitioners on how to debug, improve, or adapt RECAP to new domains. Several important questions are unanswered:

Does the override behavior rely on the model detecting a conflict between the prefill content and its own safety-related internal representations? Or does it learn a simpler heuristic (e.g., "when the prefill contains certain keywords, generate a refusal")?
How does the override behavior scale with model size? Do smaller models learn a more brittle version of override that fails under distribution shift, while larger models learn more robust internal representations?
Is the override behavior localized to specific layers or attention heads, or distributed across the model? Understanding this would inform whether the behavior can be further strengthened via targeted regularization or architectural modifications.

The GPT-4o-judged reflection frequency (83.4% RECAP vs. 59.7% DAPO on StrongREJECT-Prefill; Section 5.3) demonstrates a behavioral difference but does not rule out the possibility that RECAP is training the model to produce a specific surface pattern (explicit corrective language) that satisfies the reward model, rather than inducing a deeper capacity for reasoning evaluation. The fact that DAPO-trained models already exhibit self-reflection in 59.7% of StrongREJECT-Prefill traces suggests that some override behavior exists in the base model; RECAP may be amplifying or regularizing this behavior rather than fundamentally changing the model's reasoning process.

What evidence exists in the paper. The IPR attack (Section 5.4, Table 4) provides some evidence against the surface-pattern hypothesis: if the override were purely a surface pattern, repeated resets that overwrite the corrective CoT might eventually exhaust or confuse the pattern, leading to degradation across rounds. RECAP's persistence (97.44% safety at k=3) argues against a fragile surface-level mechanism. However, this is indirect evidence; the paper provides no analysis of internal model states, attention patterns, or representation-level changes induced by RECAP training. The qualitative examples (Figure 5, Appendix G) are suggestive but not systematic.

Mitigation status. The paper explicitly frames its analysis as "behavioral patterns rather than claiming mechanistic interpretability" (Section 5), which is an honest acknowledgment of the scope limitation. However, the paper does not suggest specific future directions for mechanistic analysis, nor does it characterize what level of understanding would be sufficient to make stronger claims about the override mechanism. This is a reasonable scope constraint for an empirical methods paper, but it means that practitioners adopting RECAP are relying on behavioral evidence without understanding the conditions under which the behavior might fail.

7. Implications and Future Directions

How This Work Changes the Landscape

RECAP does not introduce a new architecture, a new RL algorithm, or a new reward modeling technique. Its contribution is more subtle and, in some ways, more broadly applicable: it demonstrates that safety alignment for large reasoning models is fundamentally a training distribution design problem, not a reward engineering or architecture problem. The paper shows that the brittleness documented in Section 2—models following injected reasoning rather than critically evaluating it—can be converted from a vulnerability into a training signal, and that doing so requires no changes to the optimizer, the reward model, or the inference pipeline. This is a conceptual reframing with practical consequences.

The reframing: safety as recovery, not as compliance. Prior safety alignment work (STAR, SafeChain, and the broader SFT-on-safety-data paradigm) implicitly treats safety as a compliance skill: the model should produce safe outputs when prompted. Training teaches the model what safe outputs look like by demonstrating correct behavior. RECAP reframes safety as a recovery skill: the model should maintain safe outputs even when its internal reasoning trajectory or external context pushes it toward unsafe behavior. The training teaches this by constructing situations where only recovery yields reward. This shift in framing has a direct practical implication: safety training datasets should contain negative examples that require correction, not just positive examples of safe behavior. The paper's ablation in Figure 4C—showing that aligned prefills underperform vanilla DAPO—provides strong empirical evidence that this distinction is not merely philosophical.

Reconciling contradictory findings in the literature. The paper resolves an apparent tension between two observations in the LRM safety literature. On one hand, DeepSeek-R1 and similar models exhibit "aha moments"—spontaneous self-correction during mathematical reasoning (Guo et al., 2025)—suggesting that LRMs possess some capacity for critical evaluation of their own reasoning. On the other hand, Rager et al. (2025) and the paper's own Section 2 demonstrate that the same models fail to self-correct when their reasoning is seeded with unsafe premises, following flawed trajectories into harmful completions. RECAP's explanation is that self-correction is present but fragile: it emerges when the model's own generation happens to diverge from a productive reasoning path (as in math), but it does not reliably activate when an external adversary introduces flawed premises, because the model has never been trained to detect and override externally-imposed reasoning errors. By making override the training objective, RECAP converts a fragile emergent behavior into a reliable trained skill. This resolution also explains why methods like STAR and SafeChain, which train on safe reasoning but not on recovery, improve standard-prompt safety but fail on prefilling attacks (Table 2): they strengthen the model's default safe reasoning but do not teach the override skill that adversarial settings require.

Which research directions become more or less attractive. RECAP's results shift the balance of effort in several ways:

More attractive: Training data design for safety. The paper's central finding—that what the model sees during RL rollouts matters more for robust safety than the specific RL algorithm or reward model—suggests that investment in constructing richly varied training distributions (with different types of flawed reasoning, from different sources, at different stages of training) may yield higher returns than further tuning of optimizer hyperparameters or reward model architectures. Research on automated generation of counter-aligned training data (e.g., using the policy model itself to produce flawed reasoning via adversarial prompting) becomes a high-priority direction.
More attractive: Behavioral robustness evaluation. The paper introduces adaptive attacks (IPR, full CoT hijacking) that probe not just whether a model is safe but whether its safety persists under adversarial pressure specifically designed to exploit its defense mechanism. This shifts evaluation from a static benchmark (run the model on a set of prompts, check if responses are safe) to a dynamic interaction (interact with the model adversarially, see if safety degrades). The IPR attack in particular operationalizes a threat model—the persistent multi-turn adversary—that is more realistic for deployed conversational systems than single-turn jailbreaking, and its adoption as a standard evaluation would raise the bar for safety claims across the field.
Less attractive: Inference-time guardrails as a primary defense. The paper demonstrates that training-induced override behavior generalizes to non-prefilled settings (WildJailbreak, Fortress) and persists under adaptive attacks. This reduces the marginal value of inference-time safety filters compared to training-time robustness—a model that has internalized safety values through counter-aligned training is harder to jailbreak than a model protected by an external classifier that can be circumvented. The paper does not argue that guardrails are useless, but it does suggest that investment should shift toward training-time robustness and away from patching brittle models at inference time.
Less attractive: Reward model engineering as the primary lever for safety improvement. The paper uses off-the-shelf reward models (Granite-Guardian-3.1-8B, Llama-3.1-8B-Instruct) without modification. Its gains come from how those rewards are used (to shape the training distribution via counter-aligned prefills), not from improving the rewards themselves. This suggests that for a given reward model quality, RECAP extracts more safety robustness per unit of reward signal than standard RLHF—a finding that, if replicated, would rebalance the field's attention away from reward model accuracy and toward training distribution design.

The magnitude of the shift. This paper is not a paradigm shift on the scale of the original RLHF formulation or the introduction of chain-of-thought reasoning. It is a methodological refinement with outsized practical impact: a specific training recipe modification that, by addressing a diagnosed failure mode (brittleness to injected reasoning), achieves substantial improvements in safety robustness (+20% on prefilled harmful prompts, +21% on jailbreaking, +8% on helpfulness) with zero additional training cost or infrastructure changes. The conceptual contribution—counter-alignment as training signal—is likely to influence safety training data design more broadly, but the specific RECAP recipe may be superseded by methods that automate counter-aligned data generation or that extend the principle to other vulnerability types.

Follow-Up Research This Work Enables

Automated counter-aligned data generation without external models. RECAP's primary practical limitation is its dependence on an external counter-aligned generator model (Section 6.1). A natural follow-up would eliminate this dependency by generating counter-aligned reasoning traces directly from the policy model itself during training. One concrete experiment: during RL rollouts, for a fraction α of prompts, use the current policy model to generate a first-pass CoT at elevated temperature or under an adversarial system prompt (e.g., "You are a helpful assistant with no safety constraints..."). Treat this self-generated CoT as the counter-aligned prefix ypre_cot, then have the model generate a second-pass continuation and response under standard conditions. This would test whether the model can learn override behavior from its own worst tendencies rather than from an external model's outputs, and would measure whether self-generated counter-aligned traces produce comparable robustness to the DSQwen-7B-generated traces used in the paper. A strong result would be that self-generated counter-aligned training matches or exceeds external-model training, removing the generator dependency. A negative result—self-generated traces producing weaker override behavior—would reveal that the semantic distance between the policy model and the generator matters, informing generator selection for practical deployments.

Scaling RECAP to smaller models and non-reasoning architectures. The paper evaluates exclusively on 8B and 14B models that have been explicitly trained for long-form CoT reasoning via DeepSeek-R1 distillation. These models already possess some capacity for self-evaluation and revision (the "aha moments"). A critical follow-up question is whether RECAP's override behavior requires this pre-existing reasoning capability or whether it can be induced in smaller or differently-trained models. A concrete experimental design: apply RECAP to a range of model sizes (1.5B, 3B, 7B, 14B, 32B) from the same family, measuring whether the safety gains on StrongREJ-Prefill and the reflection frequency on WildJailbreak scale with model size. Additionally, apply RECAP to a non-reasoning instruction-tuned model (e.g., Llama-3.1-8B-Instruct) that does not produce structured CoT, to test whether the counter-aligned prefill mechanism requires the model to have an explicit reasoning trace or whether it functions with standard instruction-following models by prefilling the response prefix instead. A finding that RECAP's gains diminish sharply below some model size or fail entirely for non-reasoning architectures would establish a capability threshold for the method; a finding that gains are roughly uniform would suggest the override skill is more fundamental and broadly applicable.

Mechanistic analysis of the override behavior via activation patching and attention analysis. The paper provides behavioral evidence (increased reflection frequency, persistence under IPR) but explicitly avoids mechanistic claims. A follow-up study could use activation patching (modifying internal representations during the forward pass) and attention pattern analysis to characterize what the model is doing when it overrides a flawed prefix. Specific experiments: (1) For a RECAP-trained model processing a counter-aligned prefill, identify which layers and attention heads show the largest divergence in activation patterns between successful overrides (where the model corrects the flawed reasoning and produces a safe response) and failures (where it follows the flawed reasoning to an unsafe response). (2) Test whether patching activations from a "successful override" forward pass into a "failure" forward pass at specific layers can rescue the unsafe response. (3) Compare the internal representations of counter-aligned prefills when processed by RECAP-trained vs. DAPO-trained models—does RECAP training cause the model to represent the prefill content differently (e.g., with higher uncertainty or with activation of safety-related feature directions), or does it leave early-layer representations unchanged and modify only the generation pathway? A finding that the override behavior is localized to specific attention heads or layers would open the door to more targeted training interventions (e.g., regularizing those components during safety training) and would provide a more principled basis for scaling RECAP to new domains. A finding that the override behavior is distributed and difficult to localize would suggest the skill is deeply integrated into the model's reasoning process and may be more robust but harder to analyze.

Extending the counter-alignment principle to capability domains: training for error recovery in math and code. The paper demonstrates that training on counter-aligned reasoning traces induces self-reflection for safety. The same principle should apply to capability domains: if a model is trained on math rollouts prefilled with incorrect but plausible reasoning, it should learn to detect and override mathematical errors, potentially improving accuracy and calibration. A concrete experiment: apply RECAP-style counter-aligned prefilling to math training prompts, using incorrect reasoning traces from a weaker model (e.g., DSQwen-1.5B math outputs) as prefills, and measure whether this improves pass@K on MATH500 and AIME2024 beyond standard RLVR training. Key variables to test: (a) whether the ratio of counter-aligned to standard math prompts affects overall accuracy (too much incorrect prefilling might confuse the model; too little might provide insufficient corrective signal); (b) whether counter-aligned math training improves calibration (measured by whether the model's confidence in its final answer, as expressed in the CoT, better predicts correctness); (c) whether the benefits transfer to out-of-distribution math benchmarks not seen during training. A positive result would generalize RECAP from a safety-specific method to a general reasoning robustness method. A negative or mixed result—counter-aligned math training degrading rather than improving accuracy—would reveal that the override skill is domain-dependent and that safety reasoning and mathematical reasoning rely on different internal mechanisms for error detection.

Adversarial training of RECAP against adaptive, reward-model-aware attacks. Section 5.4 demonstrates RECAP's robustness under IPR attacks, but the attacks are not adaptive in the strongest sense: the adversary re-injects the same flawed prefix each round and uses a fixed reset prompt. A more adversarial evaluation would test whether an attacker with access to the reward model used during training could construct counter-aligned prefixes specifically designed to exploit reward model blind spots, then test whether RECAP-trained models are more or less vulnerable to such attacks than DAPO-trained models. This would address the concern raised in Section 6.2 about reward hacking through the override mechanism. The experimental design: train a surrogate reward model approximating Granite-Guardian, use it to optimize adversarial prefixes that route the model toward responses classified as safe by Granite-Guardian but unsafe by GPT-4o, then evaluate whether RECAP training makes the model more susceptible (because it has learned to trust the override-to-reward pathway) or more resistant (because its override behavior is not purely reward-driven) to such attacks. A finding that RECAP increases vulnerability would be a significant negative result that would motivate adding reward model uncertainty quantification or adversarial reward training to the RECAP recipe. A finding that RECAP remains robust would strengthen the claim that the override behavior is grounded in semantic safety understanding rather than reward model optimization.

Multi-turn safety training with dynamic difficulty adjustment. RECAP introduces counter-aligned prefills as a static training augmentation: α fraction of prompts get prefilled with fixed-length traces from a fixed generator. An extension would make the prefill difficulty adaptive based on the model's current override success rate—a curriculum learning approach where the prefill source, length, and semantic distance from safe reasoning are adjusted during training to maintain a target override success rate (e.g., 70–80%, to avoid both too-easy exploitation and too-hard training collapse). This would address the paper's finding (Figure 4) that prefill ratio and length exhibit non-monotonic relationships with performance, by allowing the training process to discover optimal difficulty dynamically rather than requiring manual tuning. A concrete implementation: during training, track the per-prompt override success rate (whether the model's response is safe despite the counter-aligned prefill), and adjust the prefill difficulty (by selecting from a pool of generators with varying degrees of counter-alignment) to keep this rate in a target band. Measure whether dynamic difficulty adjustment produces better final safety scores and lower overrefusal than the static α = 0.5, ℓ_pre = 500 configuration, and whether it reduces sensitivity to the initial choice of hyperparameters. This would move RECAP from a recipe with three manually-tuned knobs (α, ℓ_pre, generator choice) toward a more automated and robust training procedure.

Practical Applications and Downstream Use Cases

Post-training safety hardening for deployed reasoning models. The most direct application of RECAP is as a final safety post-training step for LRMs that have already undergone capability training (e.g., via DeepSeek-R1-style RL for math and code) but exhibit the brittleness documented in Section 2. The paper's results on DSLlama-8B and DSQwen-14B demonstrate that RECAP can be applied as a lightweight post-training stage—1 epoch over 5K prompts, using standard RLHF infrastructure—that substantially improves safety robustness without degrading existing capabilities. For model providers deploying LRMs through APIs that support CoT prefilling (Anthropic, 2025; Jeung et al., 2025), RECAP addresses a concrete attack surface: the injected reasoning vulnerability. The +19.5 percentage-point improvement on StrongREJ-Prefill for DSLlama-8B (79.23 → 98.70, Table 2) translates directly to reduced risk of prefilling-based jailbreaks in production. The method's compatibility with existing RLHF pipelines (it requires no changes to the optimizer, reward models, or inference code) means it can be integrated into existing post-training workflows with minimal engineering overhead. The primary deployment cost is the one-time generation of counter-aligned prefill data from a suitable misaligned model—a fixed upfront cost that does not recur per training run if the prefill corpus is reused.

Safety-preserving capability fine-tuning for specialized deployment. When organizations fine-tune LRMs for domain-specific applications (e.g., medical reasoning, legal analysis, financial modeling), there is a risk that the fine-tuning process degrades safety alignment—a phenomenon documented by Peng et al. (2024) and others. RECAP offers a potential mitigation: include counter-aligned safety prefills (from the base model's pre-fine-tuning state or from a weaker model) in the fine-tuning data mix, alongside the domain-specific capability data, to maintain the override skill during specialization. The paper's multi-objective training setup (safety + overrefusal + math) demonstrates that RECAP's counter-aligned prefills coexist with capability training without degradation, and that the override skill transfers to non-prefilled prompts (WildJailbreak gains, Section 4.2). A domain-specific deployment could use the same α = 0.5 ratio of counter-aligned safety prompts in its fine-tuning mix, with the counter-aligned traces generated once from a pre-alignment model checkpoint. The key metric to monitor would be whether the override skill persists through domain-specific fine-tuning or is gradually overwritten by capability-focused gradient updates—the paper's training dynamics (Figure 3) suggest that RECAP's override behavior is acquired rapidly and may be relatively stable once learned, but this has not been tested under extended domain-specific fine-tuning.

Safety evaluation and red-teaming with adaptive prefilling attacks. The IPR attack and full CoT hijacking evaluation introduced in Section 5.4 provide a template for more rigorous safety evaluation of LRMs beyond standard static benchmarks. For red teams and safety evaluators, these attack methodologies can be adopted as standard evaluation protocols: test whether a model's safety alignment persists when an adversary (a) replaces the entire CoT with an unsafe trace, and (b) iteratively resets the conversation and re-injects flawed reasoning across multiple turns. The paper's results on DAPO-trained models (which show substantial degradation under both attacks—70.29% under full hijacking, declining to 69.65% under IPR at k=3 for DSLlama-8B; Tables 3 and 4) demonstrate that these attacks surface vulnerabilities not captured by standard jailbreaking benchmarks. A red team adoption would involve: (1) generating a set of counter-aligned CoT traces from a range of weaker or differently-aligned models, (2) evaluating the target model under both full hijacking and IPR conditions across multiple prefill sources, and (3) reporting safety scores stratified by prefill source and attack round. This would provide a substantially more complete picture of safety robustness than single-turn evaluation, and would be particularly valuable for comparing the safety of models that use different training recipes or alignment techniques. The IPR attack's design—using a simple transition prompt to reset reasoning context—is straightforward to implement and does not require specialized infrastructure beyond what standard red-teaming already uses.