Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models

🎯 Pitch

A 14B reasoning model trained via sequential, domain-by-domain RL (alignment → math → coding → software engineering) outperforms its 671B SFT teacher, DeepSeek-R1-0528, on competitive coding benchmarks and wins silver at IOI 2025. The work shows that RLHF applied before any verifiable-reward RL unexpectedly and substantially boosts downstream math and code reasoning, and that this cascaded ordering near-perfectly prevents catastrophic forgetting—earlier-domain gains are preserved or improved by later stages.

1. Executive Summary

This paper introduces Cascaded Domain-wise Reinforcement Learning (Cascade RL), a post-training framework that sequentially applies reinforcement learning across distinct domains—human-feedback alignment, instruction following, mathematics, competitive programming, and software engineering—rather than blending heterogeneous prompts into a single stage. Using Qwen3-8B-Base and Qwen3-14B-Base as starting points, the authors develop Nemotron-Cascade models that operate as unified reasoners in both instruct and deep-thinking modes, with the 14B-Thinking variant outperforming its SFT teacher DeepSeek-R1-0528 (671B) on LiveCodeBench v5/v6/Pro and achieving silver-medal performance at IOI 2025. The paper demonstrates that RLHF applied as a pre-step boosts reasoning capability far beyond mere preference optimization, and that subsequent domain-specific RLVR stages rarely degrade earlier-domain performance—establishing that catastrophic forgetting is structurally mitigated in this cascaded paradigm, though the gains are bounded by problem difficulty (hard problems remain essentially unsolved across all methods).

2. Context and Motivation

The Core Problem: Unified Reasoning Models Are Hard to Build

The paper addresses a specific, practical challenge in contemporary LLM development: how do you build a single general-purpose reasoning model that excels across many domains—math, competitive programming, software engineering, instruction following, and aligned conversation—without the engineering nightmare of blending heterogeneous training data and reward signals into one monolithic RL stage?

Since the release of OpenAI o1, the field has largely bifurcated into two model categories: thinking models that spend substantial tokens reasoning before answering (e.g., DeepSeek-R1, Kimi-K2-Thinking) and instruct/non-thinking models that produce instant responses (e.g., DeepSeek-V3, GPT-4.5). The paper identifies this bifurcation as a real practical cost. Maintaining separate models complicates release pipelines, doubles deployment infrastructure, and fragments capability—an instruct model cannot deeply reason about a math problem, and a dedicated thinking model wastes tokens on simple queries. The stated goal (Section 1) is a single "unified reasoning model" that toggles between thinking and non-thinking modes at the user's discretion, per conversational turn.

However, previous attempts to build such unified models have encountered a specific failure mode that the paper documents with concrete examples (Section 1):

The degradation problem. When a unified model is trained on both thinking and non-thinking data, its reasoning performance in thinking mode often degrades relative to a dedicated thinking-only model. The authors cite explicit cases:

"although the Qwen3 series was initially released as a set of unified reasoning models, it was later reverted to separate thinking and instruct variants, with the dedicated thinking models significantly outperforming the unified models in thinking mode"

This is not a minor regression—it represents a fundamental tension. Teaching the model to answer concisely (instruct mode) and to reason verbosely (thinking mode) from the same weights appears to create interference. GPT-5's release circumvents rather than solves the problem: it routes between two specialized models via a router, which technically achieves the user-facing goal of a single endpoint but preserves the dual-model infrastructure underneath. DeepSeek-V3.1 achieves thinking-mode performance comparable to DeepSeek-R1-0528, but the authors note the technical details "have not been disclosed"—and crucially, DeepSeek-V3.1 and DeepSeek-R1-0528 are based on different base models with "likely different data blends," making the result impossible to replicate or assess.

The paper positions itself against two distinct shortcomings in the state of the art:

No open recipe exists for unified reasoning models. The community has open-weight dedicated thinking models (DeepSeek-R1 distillations, Qwen3-Thinking) and open instruct models, but the process of integrating both capabilities into one checkpoint with disclosed training data and methodology is absent. The paper aims to fill this gap explicitly: "we focus on developing an open post-training recipe" using publicly available Qwen3 base models.
Joint-RL training introduces substantial engineering complexity with no proven benefit. The dominant paradigm—exemplified by DeepSeek-R1 and Qwen3—blends diverse prompts from all domains into joint RL stages. The paper argues this introduces heterogeneity that is not just an inconvenience but a systematic drag on performance:

"Such variability complicates the RL infrastructure, slows down training, and makes training curriculum (e.g., response length extension) and hyperparameter selection more challenging."

The heterogeneity is not superficial—it is structural. Math RL uses fast symbolic verification (milliseconds per rollout); code RL uses execution-based verification that can take orders of magnitude longer. RLHF uses a trained reward model with unbounded scalar outputs; instruction-following uses binary constraint satisfaction checks. Forcing these into one training loop means the infrastructure must accommodate the slowest verification (code execution), the most memory-intensive prompts (SWE with repository context), and the most unstable reward signal (RLHF with model-based rewards) simultaneously. The worst-case governs the entire pipeline.

Why This Problem Matters

The paper's framing—and the evidence it marshals—suggests the significance is both practical and conceptual.

Practically: If a cascaded domain-wise approach can match or exceed joint training while simplifying infrastructure and curriculum design, it changes how engineering teams should allocate resources. The paper reports concrete results that would matter to a production team: a 14B model outperforms its 671B teacher on competitive coding benchmarks (Section 5), an 8B unified model closes the gap to an 8B-Thinking dedicated variant (Table 8), and RLHF—a stage often treated as purely for alignment—"boosts the model's reasoning ability far beyond mere preference optimization" (Section 1). These are not marginal gains; they suggest a fundamentally more efficient training paradigm.

Conceptually: The paper challenges a default assumption in the RL-for-LLMs literature: that training on diverse data jointly—in the style of large-scale pretraining—is necessary or optimal. Staged training in supervised learning is well-established (curriculum learning, progressive fine-tuning), but the RL setting introduces a confounding factor: catastrophic forgetting, where sequential training on different distributions overwrites previously learned behaviors. The paper's argument that Cascade RL is resistant to catastrophic forgetting (Section 4.1.1) is therefore a specific theoretical claim about the dynamics of RL—not SFT—for LLMs. If true, it would mean the field has been unnecessarily conservative in sticking to joint RL training, constrained by intuitions from supervised learning that do not transfer.

The observation that RLHF improves reasoning performance is also significant because it inverts the typical framing. RLHF is usually positioned as an alignment step that constrains or degrades raw capability in exchange for helpfulness and harmlessness. Finding that it enhances reasoning—by reducing verbosity and repetition, thereby making thinking more token-efficient (Section 4.3.2)—reframes the relationship between alignment and capability as potentially synergistic rather than purely a tradeoff.

Where Prior Approaches Fall Short

The paper identifies specific, documented limitations in existing paradigms:

Joint RL training (DeepSeek-R1, Qwen3 style). The standard approach follows a two-stage RL process: initial reasoning-oriented RL on math/code/science, then a second stage covering all domains with blended prompts. The paper argues this creates at least three concrete problems (Section 8.1):

Infrastructure bottleneck: The slowest verification (code execution, SWE Docker environments) determines the step time for the entire training loop, even though math verification is orders of magnitude faster. This means math RL, which could iterate rapidly, is artificially slowed by code prompts in the same batch.
Curriculum design difficulty: Different domains require different response-length extension schedules. Math reasoning benefits from gradually extending from 24K → 32K → 40K tokens (Section 4.5.2), while RLHF benefits from a short 12K budget to suppress verbosity. A single joint stage cannot simultaneously optimize for both.
Hyperparameter fragility: The paper shows (Section 5.2, Figure 6) that Code RL is sensitive to sampling temperature, with temperature 1.0 yielding better exploration but risking entropy explosion, while temperature 0.6 yields stable but suboptimal code accuracy. Joint training would require a single temperature setting for all domains, inevitably leaving some domains suboptimal.

Off-policy distillation from large teachers. A popular alternative to RL is to simply distill a large teacher model (e.g., DeepSeek-R1-0528, 671B) into a smaller student via supervised fine-tuning on the teacher's outputs. The paper acknowledges this approach but positions its own models against it directly: after full Cascade RL, Nemotron-Cascade-14B-Thinking surpasses DeepSeek-R1-0528—its SFT teacher—on LiveCodeBench v5/v6/Pro (Table 9). This is a specific empirical claim that RL can outperform the teacher rather than merely approximating it. The mechanism is that RL allows the model to explore solutions beyond what the teacher generated, finding better strategies for problems the teacher could not solve.

Separate instruct and thinking model pipelines. The Qwen3 approach (initially unified, later split) and GPT-5's routing approach represent the practical status quo. The paper argues these are not just inelegant but actively lose capability—the Qwen3 unified models underperformed their dedicated thinking counterparts, demonstrating that naively training on both modes degrades reasoning. The paper's unified 8B model, by contrast, closes this gap (Table 8), achieving comparable performance to the dedicated 8B-Thinking while substantially outperforming it on instruction-following (IFEval: 90.2 vs. 83.7 for dedicated thinking).

Reward-model-based reinforcement learning for math reasoning. Prior attempts to use learned reward models (rather than verifiable symbolic rewards) for math RL have had "limited success due to the inherent challenges of reward modeling in the mathematical domain" (Section 8.1). The paper cites works like Math-Shepherd and related process reward model efforts that struggle with reward hacking and distribution shift. The Cascade RL framework sidesteps this entirely by using symbolic verification for math, execution-based verification for code, and reward models only for RLHF and instruction-following—where verifiable rewards are not available. This domain-appropriate reward assignment is possible because the stages are separated.

How This Paper Positions Itself

The paper constructs its narrative around a central technical insight: sequential RL across domains does not cause catastrophic forgetting in practice, despite theoretical reasons to worry about it. Section 4.1.1 provides four specific structural arguments for why this is the case:

Policy-dependent data distribution: In RL, the model generates its own training experience. When a new domain is introduced, "old behaviors are continuously sampled if they remain useful or high-reward." This is unlike SFT, where training data from previous domains disappears from the distribution entirely.
Reward optimization, not distribution matching: RL optimizes for expected cumulative reward, not exact token-level targets. "Old knowledge that remains reward-relevant naturally persists." The model is not forced to match a new data distribution per token—it only needs to improve its reward, and behaviors from prior domains that remain reward-separating (e.g., reducing verbosity helps in all domains) are reinforced rather than overwritten.
Overlapping reward structures: "The reward structures of RLHF and RLVR overlap substantially across domains, such as math, code, reasoning, and instruction following, since they all aim to make outputs better, more accurate, and more aligned." Reducing hallucinations or maintaining conciseness benefits all domains simultaneously.
Disjoint prompt design: The authors explicitly minimize prompt overlap between stages. Math and competitive programming prompts are excluded from RLHF training (Section 4.3.1); the cascade proceeds from general domains (RLHF) to specialized ones (math → code → SWE), preventing specialized capabilities from being overwritten by generic later training.

The paper additionally positions itself as providing the missing systematic study of RLHF-RLVR interaction (Section 8.1):

"we systematically investigate the interplay between RLHF and RLVR—a topic that has been underexplored in existing literature."

Prior works like DeepSeek-R1 applied RLHF after reasoning RL, or blended them, but none had isolated the causal effect of RLHF on downstream reasoning performance. The paper's ablation in Section 6.1 (Figure 9) shows that RLHF training on a mixture of thinking and non-thinking data (the "Half-Half" setting) yields better ArenaHard and better math/code performance than thinking-only RLHF, establishing that non-thinking training has positive cross-mode transfer to reasoning tasks. This is a specific, non-obvious finding that the cascaded framework enables by making each stage's contribution individually measurable.

Finally, the paper explicitly positions itself as an open-recipe contribution in a field where frontier models (GPT-5, DeepSeek-V3.1) have not disclosed training details. The authors "transparently share our training and data curation recipes, and release the full collection of models and training data." This is not a side note—it addresses the specific gap that DeepSeek-V3.1's unified reasoning success is unreproducible because the technical details are unknown. By using publicly available Qwen3 base models and documenting every stage—SFT data blends, RL hyperparameters, reward function designs, training step counts—the paper enables direct replication and extension by the community.

3. Technical Approach

3.1 Reader Orientation

This paper builds a multi-stage post-training pipeline that takes a pretrained base language model (Qwen3-8B-Base or Qwen3-14B-Base) and produces a unified reasoning model capable of both concise instruction-following and deep chain-of-thought reasoning across math, competitive programming, software engineering, and general conversation. The core problem it solves is that naïvely training on all these domains simultaneously—the standard approach—introduces massive engineering complexity from heterogeneous reward signals (fast symbolic math verification vs. slow code execution vs. unbounded reward model scores), conflicting response-length requirements (short for alignment, long for math reasoning), and domain interference that degrades specialized capabilities. The solution's shape is a sequential cascade: apply reinforcement learning one domain at a time, ordered from general to specific, with each stage building on the previous without catastrophic forgetting.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a fixed pipeline, with information flowing left to right:

Supervised Fine-Tuning (SFT) Stage — The pretrained Qwen3-Base model is fine-tuned on a massive curated corpus spanning general conversation, math reasoning, code reasoning, science reasoning, tool use, and software engineering. This produces an SFT checkpoint that has foundational capabilities in both thinking and non-thinking modes.
Reward Model (RM) Training — An independent 72B-parameter reward model is trained on human preference data to produce scalar quality scores. This RM is used only during the RLHF stage (Step 3) and never during later verifiable-reward stages.
Cascade RL Pipeline — Starting from the SFT checkpoint, the model undergoes five sequential RL stages, each using a different reward mechanism appropriate to that domain:
- RLHF (Reinforcement Learning from Human Feedback): Uses the scalar RM scores to improve helpfulness, reduce verbosity, and align with human preferences. Operates on general-domain prompts with no math or code content.
- IF-RL (Instruction-Following RL): Uses rule-based binary verifiers that check constraint satisfaction (e.g., "response contains exactly 3 paragraphs"). Combined with RM scores for quality.
- Math RL: Uses symbolic rule-based verification (extracting \boxed{} answers and comparing to ground truth). Employs a 3-stage length extension curriculum (24K → 32K → 40K tokens).
- Code RL: Uses execution-based verification (running generated code against unit tests). Focused on competitive programming problems.
- SWE RL: Uses a hybrid execution-free reward combining lexical patch similarity and LLM-based semantic similarity. Focused on software engineering repository-level bug fixes.
Training Algorithm (GRPO) — All RL stages use Group Relative Policy Optimization with strict on-policy training, no KL divergence penalty, and token-level loss. The algorithm generates a group of rollouts from the current policy, computes group-normalized advantages, and performs a single gradient update.
Unified Chat Template — A simplified ChatML template with /think and /no_think flags appended to each user turn, enabling per-conversation-turn mode switching. Tool calling is supported via <tools> blocks in the system prompt.

The flow: Pretrained Base → SFT (multi-stage, thinking + non-thinking data) → RLHF (general alignment, reduces verbosity) → IF-RL (constraint following) → Math RL (length extension, symbolic verification) → Code RL (execution-based verification) → SWE RL (repository-level patching) → Deployable Unified Model.

3.3 Roadmap for the Deep Dive

First, the supervised fine-tuning recipe (Section 3): the two-stage curriculum, the chat template that enables unified thinking/non-thinking control, and the massive data curation effort across five distinct domains. This establishes the baseline capabilities that RL will amplify.
Second, the reward model for RLHF (Section 4.2): how preference data is constructed—including a clever off-topic prompt trick—how the Bradley-Terry model is trained, and the empirical finding that larger reward models (72B) are more robust to distribution shift and reward hacking than smaller ones (7B–32B).
Third, the GRPO training algorithm and why it is configured as strict on-policy REINFORCE (Section 4.1.2): the removal of KL regularization, token-level loss, and the specific advantage normalization that makes this work.
Fourth, the five individual RL stages in detail (Sections 4.3–4.7): each stage's reward function, data curation, hyperparameters, and the specific design choices that prevent interference with prior stages. This is where the paper's central claim—that sequential RL does not cause catastrophic forgetting—is substantiated through careful prompt exclusion, reward design, and curriculum ordering.
Fifth, the SWE-specific enhancements (Section 7): the Agentless framework decomposition into localization, repair, and validation; the execution-free reward model based on Kimi-Dev-72B semantic similarity; and the test-time scaling strategy that uses best-of-k selection with majority voting.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that sequentially applying reinforcement learning across domains—ordered from general to specific, with domain-appropriate reward signals—yields better performance and simpler infrastructure than the standard approach of blending all domains into joint RL stages. The key mechanisms are: (1) a two-stage SFT curriculum that establishes dual thinking/non-thinking capability, (2) a large 72B reward model that provides stable RLHF signals without reward shaping tricks, (3) bare-bones GRPO with strict on-policy training and no KL penalty, (4) domain-specific reward functions used sequentially without catastrophic forgetting, and (5) a unified chat template that enables per-turn mode control.

Supervised Fine-Tuning: The Two-Stage Curriculum

The SFT stage is not merely a preliminary warm-up—it is the foundation that determines the model's capacity to absorb RL improvements. The paper designs a two-stage curriculum that progressively extends context length and introduces specialized skills.

Stage 1 (16K context). The first stage trains on general-domain data plus math, science, and code reasoning data, all capped at 16,384 tokens. The key design choice is that general-domain prompts contain parallel responses in both thinking and non-thinking modes—the model sees the same prompt answered in both styles, learning that the /think and /no_think flags correspond to different response behaviors. Math, science, and code data, in contrast, contain only thinking-mode responses, since these tasks inherently benefit from extended reasoning. Training runs for one epoch.

Stage 2 (32K context). The second stage extends the maximum sequence length to 32,768 tokens and introduces new capabilities: tool use and software engineering. The data blend recombines general-domain data with new, longer math/science/code reasoning data (generated by the stronger DeepSeek-R1-0528 teacher instead of the original DeepSeek-R1), plus tool-calling and SWE datasets. All non-general domains use thinking-mode responses exclusively. Training runs for one epoch.

Why two stages? The paper does not explicitly state the rationale, but the structure implies a deliberate capacity-building progression. Stage 1 establishes basic conversational ability and reasoning patterns at moderate lengths. Stage 2 then pushes the model to handle longer reasoning chains (up to 32K) and teaches entirely new skills (tool calling, repository-level code fixes) that would be difficult to learn simultaneously with basic instruction-following. The 16K → 32K progression also serves as implicit context-length curriculum learning, preparing the model for the even longer contexts needed during RL (up to 40K for Math RL, 56K for Code RL).

The Unified Chat Template

The chat template is implemented using the standard ChatML format, with two critical additions that distinguish it from prior work:

Per-turn control flags. The /think and /no_think flags are appended to each individual user prompt, not placed in the system prompt. This is a deliberate departure from Bakouch et al. (2025), which places the flags in the system prompt and therefore controls the entire conversation globally. The per-turn design "supports both global and local control: appending the same flag to every user turn enforces consistent global behavior, while varying the flags across multi-turns enables dynamic switching within a single conversation."

No redundant mechanisms. In contrast to Qwen3, which employs "a redundant mechanism that enables mode switching in two ways: either through explicit flags or by modifying the template via the enable_thinking argument," the paper's design eliminates the template-based implicit signaling entirely. The authors' early experiments showed that "explicit flags result in more reliable mode transitions than template-based cues." The empty thinking response block—which Qwen3 uses to indicate non-thinking mode—is omitted entirely.

Tool calling extension. For tool-calling tasks, available tools are specified in the system prompt within <tools> and </tools> tags. The model is instructed to produce tool calls enclosed in <tool_call> and </tool_call> tags. On average, each conversation includes 4.4 available tools. All tool-calling data is formatted in thinking mode, meaning the model reasons about which tool to call before producing the call itself.

General-Domain SFT Data Curation

The general-domain corpus is the largest and most complex SFT component, comprising 2.8M samples with 3.2B tokens. The curation addresses three specific challenges that arise when combining diverse open-source datasets:

Challenge 1: Response brevity. Many open-source datasets contain excessively brief responses (single-word or minimal-sentence outputs). Challenge 2: Quality variance. Accuracy and style vary dramatically across data sources. Challenge 3: Stylistic inconsistency. Different labeling conventions produce incompatible response formats.

The solution: teacher-generated parallel responses. For each prompt, the authors generate responses using DeepSeek-R1-0528 (for thinking mode) and DeepSeek-V3-0324 (for non-thinking mode), with a max sequence length of 16K tokens. This ensures all responses share consistent style and quality. For prompts with verifiable ground-truth answers (e.g., multiple-choice questions), incorrect teacher responses are discarded. For prompts without ground truth, "we cross-validate the generated responses using an auxiliary model (Qwen2.5-32B-Instruct) to filter out potentially low-quality generations."

Data augmentation for scarce domains. For instruction-following and creative writing—domains where high-quality data is naturally limited—"we generate multiple responses for each prompt using different random seeds, thereby enriching diversity and improving generation quality." Multi-turn conversational data is constructed artificially: single-turn creative writing samples get a second turn instructing the model to rewrite or edit its previous response under specific constraints, and "we randomly concatenate single-turn samples to construct multi-turn conversations, emulating real-world chatbot interactions."

Knowledge-intensive subset. A dedicated 1.2M-sample, 1.5B-token subset covers professional domains including law, ethics, and other knowledge-intensive tasks. Questions are sourced from public datasets (Longpre et al., 2023; Khot et al., 2020) and augmented with domain-specific questions from "challenging areas such as professional law and ethics."

Math Reasoning SFT Data

The math SFT data follows a similar teacher-distillation pattern but with critical domain-specific quality controls.

Stage 1 sources. Prompts come from AceReason-Nemotron-1.1, which aggregates AceMath, NuminaMath, and OpenMathReasoning. Responses are generated by the original DeepSeek-R1, not the later 0528 variant. The total is 353K unique prompts with multiple responses each, yielding 2.77M samples at an average of 7.8 responses per prompt. The max context length is 16,384 tokens (16K), and "we filter out samples exceeding this limit to prevent response truncation."

Stage 2 sources. The teacher is upgraded to DeepSeek-R1-0528, which produces "longer and more detailed reasoning trajectories, leading to improved performance on challenging problems." The max context length doubles to 32,768 tokens. Crucially, the prompt set is filtered: "we filter out relatively easy questions, specifically those whose DeepSeek-R1 responses contain fewer than 2K tokens." This ensures Stage 2 focuses the model's capacity on harder problems that genuinely need extended reasoning. The result is 163K prompts generating 1.88M samples at 11.5 responses per prompt.

Decontamination. All math data undergoes 9-gram overlap filtering against standard math benchmark test sets (AIME 2024/2025, MATH, etc.).

Code Reasoning SFT Data

Code SFT combines multiple open-source competitive programming datasets:

Stage 1 (16K). Prompts from TACO, APPS, OpenCoder-Stage-2, and OpenCodeReasoning, deduplicated to 172K unique prompts. DeepSeek-R1 generates 1.42M samples (8.3 responses per prompt on average). Again, 9-gram filtering removes test set contamination.

Stage 2 (32K). Prompts are drawn from OpenCodeReasoning (challenging coding prompts) and OpenCoder-Stage2 (coding tasks with starter code entry points). The teacher is upgraded to DeepSeek-R1-0528, producing 1.39M samples from 79K prompts at an average of 17.6 responses per prompt. The substantially higher response count per prompt (17.6 vs. 8.3) indicates that the Stage 2 problems are more diverse in their solution approaches, warranting more extensive sampling.

Science Reasoning SFT Data

The science data curation involves a novel synthetic generation step:

Source filtering. Prompts come from S1K and Llama-Nemotron post-training datasets. The authors explicitly "exclude samples where models focus on analyzing each option rather than directly solving the problem and determining the correct answer"—a quality filter specific to multiple-choice science questions where the model might learn to analyze distractors rather than reason toward the answer.

Synthetic question generation. "To enrich the dataset with less common and more diverse question types, we leverage DeepSeek-R1-0528 to generate rarer questions from each given prompt, following the synthetic question generation strategy used in Liu et al. (2024c)." This step creates variant questions that test the same scientific concepts but with different surface forms, increasing the model's robustness.

Scale. 226K science prompts generate 289K Stage-1 samples (16K tokens, DeepSeek-R1) and 345K Stage-2 samples (32K tokens, DeepSeek-R1-0528). The Stage-2 science data is upsampled by 2× before blending into the full Stage-2 SFT dataset, indicating the authors found science reasoning to be underrepresented relative to math and code.

All science data is in thinking mode only.

Tool Calling SFT Data

The tool calling dataset comes from Llama-Nemotron-Nathawani et al. (2025) and is designed to cover diverse interaction patterns:

Single-turn, multi-turn, and multi-step tool interactions.
Scenarios requiring clarification questions before tool calls.
Cases where no suitable tool exists in the provided list.
Conversations averaging 4.4 available tools each.

Responses are generated by Qwen3-235B-A22B, not the DeepSeek teachers used for other domains. The dataset comprises 310K conversations with 1.41M user-assistant turns, all formatted in thinking mode. Tool calling data is used only in Stage 2 SFT.

Software Engineering SFT Data

The SWE task is decomposed into three sub-tasks using the Agentless framework (Xia et al., 2024) with a simplified variant similar to Agentless Mini (Wei et al., 2025):

Agentless framework decomposition. Rather than having the LLM autonomously plan actions and operate tools, the framework decomposes SWE task into three explicit stages:

Localization: Given the GitHub issue and repository structure (folder/file names), identify which files likely contain bugs.
Repair: Given the issue description and the contents of localized files, generate diff-style code patches that fix the identified bugs.
Patch Validation: Execute regression tests, generate reproduction tests, and use majority voting to select the most reliable patch from among generated candidates.

The simplified variant "streamlines the localization process to focus solely on identifying relevant issue files" rather than the original three-stage hierarchical strategy (file-level → class/function-level → line-level). The paper argues this "allows the LLM to dedicate more reasoning capacity to the repair task itself."

Data sources. Four open-source SWE datasets are used:

SWE-Bench-Train: the training split of the standard SWE-bench evaluation set (without human verification).
SWE-Fixer-Train: Python repositories with >100 pull requests, yielding 115K instances after heuristic filtering.
SWE-reBench: 21K+ interactive Python-based SWE tasks constructed through an automated pipeline.
SWE-Smith: 50K synthetically generated instances from 128 GitHub repositories, built by automatically injecting bugs into codebases.

Deduplication strategy. "We exclude all instances originating from repositories present in the evaluation dataset" (SWE-bench Verified). Additionally, cross-source deduplication matches both repository names and base commit identifiers.

Response generation. DeepSeek-R1-0528 generates responses for three sub-tasks:

Code localization: The model outputs a prioritized list of file names ranked from most to least likely to contain bugs.
Code repair: The model outputs SEARCH/REPLACE edit blocks wrapped in a specific diff format. The repair prompt includes the problem statement and the concatenated contents of localized code files with surrounding context (imports, class definitions, dependent functions).
Test code generation: The model outputs unit tests and reproduction tests designed to verify both bug replication and patch correctness.

Response filtering. Different sub-tasks use different filtering criteria:

Localization: Retain only samples where the model identifies all ground-truth buggy files (recall = 1.0).
Repair: Use the Unidiff library to compute lexical similarity between generated and ground-truth patches. A stratified approach: instances with ≥4 out of 8 responses exceeding 0.5 similarity go to SFT; instances with ≥1 out of 8 achieving non-zero similarity (but fewer than 4 exceeding 0.5) are reserved for SWE RL; SWE-Fixer-Train instances exceeding 0.5 similarity are included regardless.
Test code generation: Retain only trajectories that parse and execute without syntax errors.

Final dataset composition. The Code Repair dataset contains 127K instances: 17K from SWE-Bench-Train, 17K from SWE-reBench, 18K from SWE-Smith, and 77K from SWE-Fixer-Train. Localization comprises 92K samples; test case generation comprises 31K samples. All SWE datasets are upsampled by 3× in the Stage-2 SFT blend. These SWE data are all formatted in thinking mode.

The repair prompt design. The repair prompt concatenates multiple localized files and their surrounding code snippets (e.g., imports, class definitions, dependent functions) into a unified context, enabling the model to reason over repository-level dependencies. The output format requires specific syntax: each SEARCH/REPLACE edit must use the diff format with <<<<<<< SEARCH, =======, and >>>>>>> REPLACE markers. The model is explicitly instructed to produce targeted patches, not full file rewrites, to "reduce hallucinations and syntactic errors."

Post-SFT Results and the Baseline for Cascade RL

The SFT stage produces three model variants: 8B-Thinking SFT, 8B unified SFT, and 14B-Thinking SFT (Table 2). Key observations:

The 8B unified model performs on par with the dedicated 8B-Thinking model on reasoning benchmarks while surpassing it on IFEval (70.8 vs. 66.3), a task naturally suited to instruct mode. This is notable because both models were trained on the same thinking data, but the unified model additionally incorporated non-thinking data without losing reasoning capability.
The 14B-Thinking SFT model is uniformly stronger than both 8B variants, with AIME 2024 at 86.9 (vs. 83.6–83.8), LCB v5 at 66.1 (vs. 59.2–59.6), and SWE-bench at 34.5 (vs. 26.1–30.2).

These SFT checkpoints serve as the initialization for the Cascade RL pipeline. The paper emphasizes that "the SFT and RL datasets are strictly disjoint in terms of prompts, so the model cannot leverage memorized answers for given prompts from SFT during RL training."

Reward Model Training for RLHF

The reward model (RM) provides scalar quality scores during the RLHF stage. Its training is independent of, and prior to, the Cascade RL pipeline.

Preference data composition. The RM training dataset is a mixture of open-source and in-house data totaling 82K preference pairs:

HelpSteer2: 10K human-annotated pairs with multi-aspect annotations (helpfulness, correctness, coherence, complexity, verbosity). Each sample has a scalar preference score ranging from −3 to 3, where negative values indicate the first response is better and positive values indicate the second response is better. Samples with a score of 0 (tied quality) are filtered out, leaving 36K pairs.
Synthetic "bad responses from strong models" data: The paper describes a specific technique inspired by Park et al. (2024): construct preference pairs where the bad response comes from a stronger LLM and the good response comes from a weaker one. The intuition is that strong models' mistakes are more subtle and therefore harder for the RM to distinguish—making the training signal more informative. The method works as follows:
1. DeepSeek-V3 generates an off-topic prompt by rewriting the original prompt: "The generated input is highly relevant to but different from the given input. The correct response to the generated input superficially resembles the correct response to the given input as much as possible. But actually, the correct response to the generated input should not be a correct response to the given input."
2. DeepSeek-V3 (the "strong" model) answers this off-topic prompt, producing a response that is superficially plausible but incorrect for the original prompt.
3. DeepSeek-V3-0324 (the "weaker" model) answers the original prompt correctly.
4. An auxiliary model verifies that the off-topic prompt genuinely differs from the original by asking a yes/no question: "Are the two instructions asking the same thing?"
The result is a preference pair where the dispreferred response comes from a stronger model, making the discrimination task harder and more informative.

The authors also "tried explicitly instructing strong LLMs to produce subtly erroneous answers to the given prompts; however, this approach was unsuccessful." This is a notable negative result—LLMs struggle to deliberately generate plausible-but-wrong answers on command, but will naturally produce them when given a slightly misdirected prompt.

Training objective. The RM is trained using the standard Bradley-Terry preference model:

$\mathcal{L}(\theta) = \mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}} \left[ \log P_{\text{BT}}(y^+ \succ y^- \mid x) \right]$

where

$P_{\text{BT}}(y^+ \succ y^- \mid x) = \frac{\exp(r_\theta(x, y^+))}{\exp(r_\theta(x, y^+)) + \exp(r_\theta(x, y^-))}$

where $r_\theta(x, y)$ is the scalar reward predicted by the RM for response $y$ given prompt $x$ , $y^+$ is the preferred response, and $y^-$ is the dispreferred response.

What it computes: the negative log-likelihood of the observed human preferences under a model that assigns exponentially higher probability to preferred responses. For each preference pair $(x, y^+, y^-)$ , the model's scalar scores are converted to a probability that $y^+$ is preferred via the softmax over both scores. The loss then maximizes the log of this probability. The expectation is taken over the training distribution $\mathcal{D}$ of human-annotated preference pairs.

Why this form: the Bradley-Terry model is the standard statistical model for pairwise comparisons. It assumes that each response has a latent "quality" score, and the probability of preferring one response over another is proportional to the exponential of the difference in their scores. This connects naturally to the RLHF policy gradient: during RLHF, the policy is optimized against these scores, and the exponential form ensures that small score differences (e.g., 0.1) still produce meaningful gradient signals—the softmax compresses large differences and amplifies small ones.

Model architecture and training configuration. The RM is initialized from Qwen2.5-72B-Instruct with a linear prediction head on top of its last hidden layer. Training hyperparameters: batch size 256, learning rate 2e-6, AdamW optimizer, 1 epoch. The authors note that "longer training schedules" were tested but "a single epoch yielded the best results."

RM backbone ablation. The paper conducts a systematic study of RM size and initialization (Table 3):

Scaling law holds: Qwen2.5-Instruct-7B → 14B → 32B → 72B improves RewardBench from 91.98 → 93.22 → 93.56 → 95.15 overall.
Large-scale preference pretraining (WorldPM-72B, a Qwen2.5-72B further trained on 15M diverse preference pairs) provides faster convergence but is slightly outperformed by vanilla Qwen2.5-72B-Instruct with extended training (94.08 vs. 95.15).
Reasoning models (Qwen3-14B) perform dramatically worse as RM backbones than instruct models of the same size: 46.38 for Qwen3-14B thinking mode vs. 93.22 for Qwen2.5-Instruct-14B. Even with non-thinking mode enabled, Qwen3-14B (91.72) fails to match Qwen2.5-Instruct-14B (93.22). The hypothesis: "Qwen3 reasoning models are primarily optimized for reasoning-centric tasks (e.g., math and code) rather than general human preference alignment."

Key insight on RM size and reward hacking. The RM size ablation reveals a practically crucial finding that larger RMs are qualitatively different, not just quantitatively better:

"Larger LLMs demonstrated greater robustness to stylistic artifacts in the preference data, whereas smaller models tended to focus more on the style of a response rather than its overall quality."

This stylistic bias manifests concretely: when RLHF is performed with a 7B RM, the policy learns to hack the reward by increasing response length rather than improving content quality. Under style-controlled ArenaHard evaluation (which normalizes for length and formatting), the 7B RM-trained policy shows a large performance drop, while the 72B RM-trained policy does not. This is direct evidence that smaller RMs confound quality with style, and that using them for RLHF will produce policies that game superficial features.

The GRPO Training Algorithm

All five Cascade RL stages use the same underlying algorithm: Group Relative Policy Optimization (GRPO) with specific modifications that make it equivalent to group-normalized REINFORCE.

The GRPO objective with removed KL term:

$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{(q, a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_\theta(\cdot \mid q)} \left[ \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \hat{A}_{i,t} \right]$

where

$\hat{A}_{i,t} = \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)} \quad \text{for all } t$

where $q$ is a prompt from the training distribution $\mathcal{D}$ , $a$ is the ground-truth answer (used only for verification in RLVR, not for loss computation), $\{o_i\}_{i=1}^G$ is a group of $G$ responses sampled from the current policy $\pi_\theta$ , $|o_i|$ is the length of the $i$ -th response in tokens, $r_i$ is the scalar reward for response $o_i$ , and $\hat{A}_{i,t}$ is the group-normalized advantage assigned to every token in response $o_i$ .

What it computes: for each training iteration, the algorithm: (1) samples a batch of prompts, (2) generates $G$ responses per prompt from the current policy, (3) computes a scalar reward for each response (either from the reward model in RLHF or from verifiable checks in RLVR), (4) normalizes these rewards within each group by subtracting the group mean and dividing by the group standard deviation, producing an advantage $\hat{A}_{i,t}$ that is positive for above-average responses and negative for below-average ones, (5) assigns this same advantage to every token in the response (token-level loss), (6) computes the policy gradient as the average of $\hat{A}_{i,t} \cdot \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t})$ over all tokens in the batch, (7) performs a single gradient update. The advantage normalization ensures that within each group, exactly half the probability mass is pushed up and half is pushed down, regardless of the absolute reward scale.

Why this form: the standard GRPO from Shao et al. (2024) includes a KL divergence penalty $-\beta D_{KL}(\pi_\theta \| \pi_{\text{ref}})$ that prevents the policy from diverging too far from a reference policy. This paper removes it entirely, setting $\beta = 0$ . The justification is empirical: "this on-policy setup contributes to stable RL training and mitigates entropy collapse." The key mechanism is that strict on-policy training ensures the importance sampling ratio is exactly 1—the policy used for data collection is identical to the one being updated, so there is no need to clip importance weights or penalize divergence. The KL term is unnecessary because the policy never sees off-policy data that could cause destructive updates.

The token-level loss (vs. sequence-level) is another deliberate choice. Standard GRPO averages losses within each sequence first, then across sequences. Token-level loss averages all token losses directly across the batch. This means longer sequences contribute proportionally more to the gradient, which the paper argues is appropriate for long-chain-of-thought reasoning: every reasoning token should receive reinforcement signal proportional to its contribution to the final outcome.

Training configuration. Throughout all Cascade RL stages, the following settings are shared unless otherwise noted:

Sampling: $G = 8$ rollouts per prompt (16 for SWE RL), temperature 0.6–1.2 depending on the stage, top-p 0.95.
Optimization: AdamW with $\beta_1 = 0.9, \beta_2 = 0.95$ .
Entropy coefficient: 0 (no entropy bonus).
KL coefficient: 0 (no KL penalty).
On-policy: one gradient update per batch of rollouts; rollouts are generated fresh from the current policy for each update.

Stage 1: Reinforcement Learning from Human Feedback (RLHF)

RLHF is the first RL stage because it provides a foundation that all subsequent stages depend on.

Why RLHF first? The paper identifies two mechanisms by which RLHF improves downstream reasoning:

Verbosity reduction: "RLHF tends to reduce repetition and verbosity, thereby compressing the number of thinking tokens for simpler questions." The SFT models, trained on teacher-generated long-form reasoning, tend to produce unnecessarily verbose chains of thought. RLHF penalizes this (the reward model prefers concise, clear responses), making the model more token-efficient. This matters immensely for later RL stages: Math RL and Code RL involve generating up to 40K–56K tokens, and wasted tokens on simple problems reduce the effective budget for complex ones.
Quality improvement: "RLHF substantially improves the quality of generated responses... thereby enhancing the reasoning performance within constrained response lengths." This is measured concretely: after RLHF (Table 4 vs. Table 2), AIME 2024 improves from 83.6 to 86.4 (8B-Thinking), LCB v5 improves from 59.6 to 70.3, and GPQA Diamond improves from 64.2 to 67.3—all despite the RLHF data containing no math or code prompts.

Data curation. The RLHF training dataset is a subset of the RM's preference data with specific exclusions: "We exclude prompts related to mathematics and competitive programming, since the reward model may not provide reward signals as reliable as those produced by rule-based or execution-based verifiers used in subsequent Math RL and Code RL stages." An early experiment found that failing to exclude math prompts caused a 2% absolute drop on AIME25. The result is a dataset focused on helpfulness, harmlessness, and general alignment.

Reward function. The reward for a response is simply the scalar output of the 72B reward model. Key implementation details:

Answer extraction: For thinking-mode responses, only the final summary after <\think> is sent to the RM; the reasoning trace is excluded. For non-thinking mode, the full assistant response is used.
Incomplete generation handling: If the thinking process does not terminate properly (missing <\think> token), the entire unfinished response is sent to the RM. "Such incomplete generations typically receive low reward scores because the reward model was not trained on unfinished or unseen reasoning traces, effectively penalizing verbose or incomplete thinking processes."
Code-switching penalty: When the prompt is purely in English but the response contains non-English tokens, an adaptive penalty is applied: the reward for mixed-language generations is set to "the lowest score in the batch minus 10, ensuring they receive the lowest relative score under the GRPO algorithm and thus the strongest penalty for code-switching behavior."

Maximum response length: 12K tokens. This is deliberately short—shorter than any subsequent RL stage (8K–56K). The rationale: "without applying overlong filtering, which effectively encourages more succinct generations." By setting a tight budget and not filtering out overlong samples (which would remove them from the gradient computation), the model learns through negative reward signals that verbosity is costly.

Training mode for unified models. For the unified 8B model, RLHF training splits each batch evenly between thinking and non-thinking modes (the "Half-Half" setting). The paper's ablation (Section 6.1, Figure 9) shows this outperforms both thinking-only and non-thinking-only RLHF on ArenaHard, AIME, and LiveCodeBench—despite all three benchmarks being evaluated in thinking mode. The mechanism appears to be cross-mode transfer: learning to be helpful and concise in non-thinking mode improves the quality of thinking-mode outputs as well.

Hyperparameters: batch size 256, learning rate 2e-6, 800 steps for 8B, 900 steps for 14B, rollout size 8, temperature 0.6, top-p 0.95. Overlong filtering: False.

Stage 2: Instruction-Following Reinforcement Learning (IF-RL)

IF-RL addresses a specific degradation observed after RLHF: IFEval scores dropped dramatically (e.g., 66.3 → 45.5 for 8B-Thinking, Table 4 vs. Table 2). The cause: RLHF's reward model encourages human-preferred qualities that can conflict with strict instruction-following constraints.

Data curation. The IF-RL dataset blends three sources:

Llama-Nemotron IF data: 56K synthetically generated prompts containing 1–10 detailed constraints from the IFEval taxonomy, reduced to 40K after "extensive preprocessing and filtering" to remove noise.
Custom data: 60K samples pairing user prompts from LMSYS-Chat-1M with instruction constraints from the IFEval taxonomy, designed to increase diversity.
IF-RLVR training data (Pyatkin et al., 2025): prompts with constraints from either IFEval or IF-Bench-Train taxonomy, with base prompts from Tulu-3-SFT. This data specifically trains robustness to unseen constraint taxonomies.

Two-stage training. The IF-RL process uses two distinct data blends with increasing difficulty:

Stage 1: Constraints from the IFEval taxonomy (seen during SFT).
Stage 2: Constraints from the IF-Bench-Train taxonomy (unseen during SFT), testing generalization.

Dynamic filtering (Yu et al., 2025) is applied in both stages to "stabilize the IF-RL training and improve the results by ensuring all prompts in the batch with effective gradients." This means prompts where all rollouts succeed or all fail are removed from the batch, preventing gradient collapse.

The interference problem: IF-RL can degrade alignment. A major challenge identified through early experiments: "naively using a rule-based IF verifier as the reward function degraded human alignment results." The mechanism is reward hacking: a response that satisfies all IF constraints but is poorly written (e.g., a low-quality summary that happens to be under 300 words) would receive a full reward of 1. The IF verifier cares only about constraint satisfaction, not quality.

Two solutions are developed for different model types:

Solution 1 (Unified models): IF-RL in non-thinking mode only. An "effective strategy emerges": apply RLHF in both thinking and non-thinking modes, then apply IF-RL only in non-thinking mode. The hypothesis: "applying IF-RL to an RLHF-trained model in the non-thinking mode is far less likely to generate low-quality responses than applying it in the thinking mode, and therefore is much less prone to reward-hacking the rule-based IF verifier." The result is that thinking-mode IF performance also improves (8B unified achieves IFEval 85.3 in thinking mode) despite IF-RL training being exclusively non-thinking.

Solution 2 (Thinking models): Combined reward function. For dedicated thinking models, IF-RL must operate in thinking mode. The reward function combines the binary IF verifier with the RM score:

$r_i = \begin{cases} R_{\text{IF}}(o_i) + \text{sigmoid}(\hat{R}_{\text{RM}}(o_i)), & \text{if } R_{\text{IF}}(o_i) = 1 \\ 0, & \text{otherwise} \end{cases}$

where

$\hat{R}_{\text{RM}}(o_i) = \frac{R_{\text{RM}}(o_i) - \text{mean}(\{R_{\text{RM}}(o_i)\}_{i=1}^G)}{\text{std}(\{R_{\text{RM}}(o_i)\}_{i=1}^G)}$

where $R_{\text{IF}}(o_i) \in \{0, 1\}$ is the binary reward from the instruction-following verifier (1 if all constraints are satisfied, 0 otherwise), $R_{\text{RM}}(o_i)$ is the raw scalar output from the same 72B reward model used in RLHF, $\hat{R}_{\text{RM}}(o_i)$ is the group-normalized version of that RM score (mean 0, std 1), and $\text{sigmoid}(\cdot)$ maps to (0, 1).

What it computes: if the response fails any IF constraint, the reward is 0—the model gets no credit regardless of quality. If all constraints are satisfied, the reward is 1 plus a quality bonus between 0 and 1 based on the RM's assessment relative to other responses in the same group. A response that satisfies constraints AND is in the top half of the group (positive normalized RM score) gets between 1.5 and 2; a response that satisfies constraints but is below average gets between 1 and 1.5.

Why this form: the binary gate at $R_{\text{IF}} = 0$ ensures the model cannot hack the reward by producing low-quality constraint-satisfying responses—the RM quality bonus is only available if constraints are met. The sigmoid transform on the group-normalized RM score ensures the quality component is on the same scale as the constraint component (0–1 range for both, so the total is 1–2 for valid responses). Group normalization is necessary because the raw RM scores are unbounded and their scale varies across prompts.

Hyperparameters: Stage 1 max response length: 8K (both unified and thinking), Stage 2: 8K (unified), 16K (thinking). Batch size 256, rollout 8, learning rate 2e-6. Overlong filtering: False in Stage 1 for both, False in Stage 2 for unified, True in Stage 2 for thinking.

Results after IF-RL (Table 5). IFEval and IFBench show massive improvements (e.g., 8B unified IFEval: 50.1 → 90.4), confirming the IF-RL works for its primary purpose. Minor degradations appear on reasoning benchmarks (e.g., AIME 2024 drops 0.8–2.8 points), but "most of them are fully recoverable and are further improved after the subsequent Math RL, Code RL, and SWE RL stages." The paper also notes that IF-RL "generally reduces model entropy and shortens the average length of reasoning tokens"—a side effect that compresses reasoning traces and makes subsequent RL stages more efficient.

Stage 3: Math RL

Math RL is the first domain where verifiable rewards (RLVR) are used exclusively, with no RM involvement.

Data curation. The training set is derived from AceReason-Math (Chen et al., 2025b) with aggressive filtering. Starting from 49K problems, the dataset is reduced to 14K through:

9-gram contamination filtering against AIME 2024/2025 and MATH.
Format exclusions: multiple-choice, true/false, proof-based, multi-sub-question, non-English, and figure-referencing problems are removed.
Noise filtering: "Because NuminaMath contains OCR and parsing errors, each problem is verified by the DeepSeek-R1 model with up to eight attempts. A rule-based verifier retains only problems with majority-voted correct answers, while ambiguous or noisy items are discarded."
Difficulty filtering: Problems where AceReason-Nemotron-7B achieves ≥75% success rate over 16 samples are removed as "overly simple" and providing no useful gradient signal.

Reward function. The reward is strictly binary based on answer correctness:

Extract the boxed answer (\boxed{}) following the <\think> token.
Verify against ground truth using the AceMath rule-based verifier: 1 for correct, 0 for incorrect.
Code-switching penalty: A reward of -1 (not 0) when non-English tokens appear in the reasoning chain. This is stricter than the RLHF penalty (which sets reward to batch minimum minus 10) because -1 is always below the binary reward range of {0,1}, ensuring mixed-language outputs are always the least preferred in the group.

The three-stage length extension curriculum. This is the most carefully engineered component of the paper, with each stage serving a distinct purpose:

Stage 1: 24K Compression Stage. The motivation: "small and medium sized SFT checkpoints tend to generate overlong reasoning chains, leading to incomplete ratios of 15–20% on the AIME benchmark under a 32K token budget. This overgeneration wastes tokens and often leaves solutions unfinished." By imposing a tight 24K budget, the model learns to compress reasoning. Overlong filtering is deliberately applied (overlong responses are skipped, not given reward 0), because "doing so may excessively penalize long reasoning on difficult problems, causing a sharp performance drop during compression." After ~100 steps, the incomplete ratio drops from 30–50% to ~15%.

Stage 2: 32K Extension Stage. Once reasoning chains are stabilized at 24K, the budget extends to 32K. Overlong filtering is turned off—"we do not apply overlong filtering to regularize reasoning length to fit the 32K context (i.e., assigning a reward of 0 for overlong generation)." This allows the model to naturally expand its reasoning when problems warrant it. The model "not only adapts to the larger budget but also begins to surpass their starting accuracy."

Stage 3: 40K Long Reasoning Stage. By the end of Stage 2, easy and medium AIME problems are nearly saturated (99% and 85% respectively), but hard problems plateau below 30%. The 40K stage "explicitly incentivizes the model to leverage more tokens during reasoning." Result: "performance on hard AIME problems improves significantly from 30 to 40%, while performance on other problems remains at high level."

The evaluation budget is 64K tokens (with YaRN scaling factor 2), so even at 40K training budget, the model has headroom at inference time.

Dynamic filtering. This is applied at the epoch level (not batch level) and serves to maintain gradient quality:

After each epoch, problems with 100% accuracy (too easy) or 0% accuracy (unsolvable) are removed.
Hard problems (0% accuracy) are re-sampled with 10% probability, since the policy may learn to solve them during subsequent updates within the same epoch.
Easy problems (100% accuracy) are re-sampled with 1% probability, to "stabilize training, as the policy may forget how to solve them within an epoch."
This ensures ~90% of training samples contribute meaningful learning signals.

The paper notes this is "a more efficient alternative to batch-based dynamic sampling, as the latter requires substantially more rollouts to construct a fixed-size batch free of overly simple or unsolvable prompts."

Hyperparameters: Batch size 128, rollout 8, learning rate 2e-6 for 8B, 2.5e-6 for 14B. Temperature: 1.0 for 8B (all stages), 1.2 → 1.1 for 14B. Each stage: 100–200 steps, depending on when clip-ratio reaches 10%.

Results after Math RL (Table 6). AIME 2024 improves to 90.2 (8B, both variants) and 90.4 (14B). AIME 2025 improves to 80.2/81.9/83.3. Code benchmarks also improve: LCB v5 for 8B-Thinking goes from 69.0 to 71.2, confirming cross-domain transfer. Knowledge and alignment benchmarks show minimal change.

Stage 4: Code RL

Code RL is applied after Math RL, which "serves as an effective warm-up that stabilizes future RL training and enhances the model's general reasoning capabilities."

Data curation. The training set is derived from AceReason-Nemotron coding corpus, which aggregates TACO, APPS, DeepCoder, and other competitive programming datasets. Filtering is exceptionally strict:

Incompatible format removal: Problems requiring interactive I/O or special judges are excluded—the verification framework expects standard output comparison.
Insufficient test coverage removal: Problems without adequate unit tests for edge/corner cases are removed. "This filtering process substantially reduces false-positive and false-negative reward signals during training, which are known to degrade Code RL performance."
Deduplication and contamination control: 9-gram filtering plus raw problem URL matching.
Difficulty calibration: AceReason-Nemotron-7B is used to remove trivial problems (solved in all 8/8 rollouts); DeepSeek-R1-0528 removes intractable problems (solved in 0/8 rollouts).
Final training set: 9.8K samples.

Reward function. A strict binary rule-based reward: 1 only when the generated code passes all test cases; 0 otherwise. The verification uses the parallelized code verifier from the AceReason Evaluation Toolkit. Asynchronous reward computation is deployed in VeRL: verification runs in parallel with generation, reducing average verification time per batch from 1172.4 seconds to 416.2 seconds (on 8 DGX H100 nodes, batch 128, rollout 8).

Code-switching penalty. Unlike Math RL (which assigns -1), Code RL assigns 0 for language mixing. The paper found empirically that "-1 negatively impacts coding performance. This is likely because the additional penalty encourages the model to produce incorrect answers without code-switching in GRPO training when all rollouts in the group are either incorrect or contain language mixing." When all 8 rollouts are wrong, the group has zero variance, and the advantage is 0 for all—but a -1 reward still shifts the baseline, causing unintended gradient effects.

Temperature sensitivity. Code RL is "sensitive to the temperature configuration." The paper's ablation (Section 5.2, Figure 6) shows that temperature 1.0 yields better final accuracy but suffers from training instability (entropy explosion), while temperature 0.6 yields stable but lower accuracy. The production training uses temperature 1.0 with monitoring for entropy collapse.

Hyperparameters: Batch size 128, rollout 8, learning rate 4e-6 (double the math RL rate), 64–90 steps depending on model variant. Maximum response length: 44K–56K. Overlong filtering: False.

Results after Code RL (Table 7). This is where the paper's strongest gains appear. LCB v5: 8B-Thinking 74.3, 8B unified 75.3, 14B-Thinking 78.0. LCB v6: 71.0, 71.5, 74.8 respectively. The 8B unified model's LCB v5 score (75.3) now exceeds DeepSeek-R1-0528 (74.8)—a 671B teacher being surpassed by its 8B student. Math benchmarks show slight fluctuations (AIME 2024 drops 1.1–1.9 points for 8B models) but remain within normal evaluation variance.

Stage 5: SWE RL

SWE RL is the final and most specialized stage, training the model to generate code patches that fix real GitHub issues.

Data curation. The RL dataset uses harder instances than SFT: prompts where fewer than 4 of 8 DeepSeek-R1-0528 responses exceed 0.5 Unidiff similarity to the ground-truth patch, but at least 1 response has non-zero similarity (confirming the problem is solvable).

Input context extension. A critical challenge: during SFT, prompts contain only ground-truth localization files. During evaluation, the model receives files from the localization stage, which may include irrelevant files. This out-of-distribution gap causes performance degradation. The solution: construct RL prompts that include both ground-truth files and noisy retrieved files:

Ground-truth only subset: Prompts with only the correct files (matching SFT distribution).
Mixed localization subset: Prompts built from files localized by DeepSeek-R1-0528 plus ground-truth files, up to 5 files total. The procedure: start with ground-truth files; if total length < $l$ tokens, add noisy files one by one until $l$ would be exceeded. File order is randomized. This teaches the model to identify relevant code among distractors.

Prompts shorter than 8K tokens are discarded (too simple). The maximum prompt length $l$ is ablated (Table 13): 16K, 24K, 32K, and 40K. Results show 24K–32K is optimal; 40K degrades performance, likely because "the pretrained Qwen3-14B-Base has limited long-context capability at 32K."

The execution-free reward model. Traditional SWE RL runs generated patches in Docker containers—accurate but slow, limiting training to ~10K instances. The paper's key innovation: a reward function that never executes code:

$r(\hat{p}, p^*) = \begin{cases} 1, & \text{if } s_{\text{lex}}(\hat{p}, p^*) = 1 \\ 0, & \hat{p} \text{ is identical to the original code snippet} \\ -1, & \text{if } \hat{p} \text{ cannot be parsed} \\ s_{\text{sem}}(\hat{p}, p^*), & \text{otherwise} \end{cases}$

where $s_{\text{lex}}(\hat{p}, p^*)$ is lexical similarity computed by the Unidiff library (comparing generated patch $\hat{p}$ to ground-truth $p^*$ line-by-line), and $s_{\text{sem}}(\hat{p}, p^*)$ is semantic similarity: the probability that Kimi-Dev-72B assigns to the token "YES" when asked whether the generated and golden patches have the same effect.

What it computes: four cases. Perfect lexical match → reward 1. No change at all (patch identical to original code) → reward 0, because the model generated a no-op. Unparseable patch → reward -1, penalizing syntactic errors. Partial match → reward equals the LLM's semantic similarity estimate (a probability between 0 and 1), capturing cases where the generated patch uses a different implementation approach but achieves the same fix.

Why this form: the piecewise design addresses three failure modes. The lexical match gate at $s_{\text{lex}} = 1$ provides a high-confidence positive signal when the generated patch exactly matches the ground truth. The no-change penalty prevents the model from producing safe but useless outputs. The unparseable penalty prevents syntactic garbage. The LLM-judge component handles the most common case: patches that differ from the ground truth in implementation but are semantically equivalent. Using Kimi-Dev-72B rather than a smaller model is motivated by the need for reliable semantic judgments on complex code changes.

Why not use Docker execution? The paper does not explicitly argue against Docker execution but implies scalability as the primary constraint: "running and managing numerous Docker instances significantly limits scalability, constraining prior work to training datasets of around 10k unique instances." The execution-free reward enables training on much larger datasets (the SWE RL training set is larger than 10K, though exact size isn't stated) at faster iteration speeds.

Ablation: lexical vs. semantic similarity (Table 12). Using semantic similarity (LLM judge) outperforms lexical similarity as the continuous reward component (cond. 4: 42.9% avg@4 vs. cond. 2: 42.8%). Reward shaping (setting rewards below 0.5 to 0) helps lexical similarity (cond. 2: 42.8% vs. cond. 1: 42.6%) but not semantic similarity (cond. 4: 42.9% vs. cond. 3: 43.0%), indicating that "semantic similarity continues to provide meaningful training signals even when code similarity is low."

Two-stage context extension curriculum. For 8B models, SWE RL uses two stages:

16K context initialization: A warmup stage because "directly initializing training with a 24K context length leads to suboptimal convergence and degraded final performance—a phenomenon we attribute to the model's initial difficulty in attending to and synthesizing information across extended sequences."
24K context extension: Once the 16K stage plateaus, context is extended to 24K. The model "has already built strong multi-file analysis skills at 16K, forming a solid basis for scaling to longer context."

For the 14B model, a single 32K stage is used—the larger model has sufficient long-context capacity from the start.

Hyperparameters: Batch size 128, rollout 16 (double other stages), learning rate 2.5e-6, temperature 1.0. Overlong filtering: True (applied when responses hit the 16K max response length). Steps: 30 + 30 for 8B, 120 for 14B.

Results after SWE RL (Table 8). SWE-bench Verified improves dramatically: 8B-Thinking 33.3 → 38.5, 8B unified 31.6 → 37.2, 14B-Thinking 39.6 → 43.1. The 8B unified model closes the gap with 8B-Thinking (37.2 vs. 38.5), achieving comparable SWE performance while maintaining superior instruction-following. All other benchmarks show minimal changes (±0.5–1%), confirming the cascade's resistance to forgetting.

The Localization Stage for SWE Evaluation

The generation–retrieval approach for code localization is used only at evaluation time (not during RL training, which focuses exclusively on code repair).

Generation-based approach: The model is prompted with the issue description and repository structure (folder/file names, not file contents). Multiple rollouts (16 by default) are aggregated by ranking candidate files by frequency of appearance across rollouts.

Retrieval-based approach: NV-Embed-Code (Sohrabizadeh et al., 2025) encodes the full source code of each repository file and retrieves files whose code content is semantically similar to the problem context. This has access to file contents, unlike the generation approach.

Fusion: The two rankings are combined using reciprocal rank fusion with $k = 0$ , meaning the fusion score for a file is $\frac{1}{\text{rank}_{\text{gen}} + 1} + \frac{1}{\text{rank}_{\text{ret}} + 1}$ . The paper notes the retrieval-based method outperforms generation-based (Figure 11), likely because "the retrieval-based approach encodes the full source code content of each repository, whereas the generation-based approach relies only on repository structure."

Test-Time Scaling for SWE

The test-time scaling (TTS) pipeline operates at inference time to improve patch selection:

Generate $k$ candidate repair patches (temperature 0.6, top-p 0.95).
Generate 40 reproduction tests per instance.
Filter candidates by regression test pass rate (removing patches that break existing functionality).
Execute a curated subset of generated reproduction tests on each surviving candidate.
Rank by combined pass rate (regression + reproduction).
Break ties by majority voting (most frequently generated patch across samples), then by minimal edit distance.

This is evaluated as best@ $k$ (whether any candidate in the top- $k$ is correct) vs. majority@ $k$ (whether the majority-voted candidate is correct). Figure 12 shows best@ $k$ substantially outperforms majority@ $k$ , and both scale monotonically with $k$ . Nemotron-Cascade-14B-Thinking achieves best@32 = 53.8%, competitive with specialized 32B SWE models.

Why Cascade RL Resists Catastrophic Forgetting

Section 4.1.1 provides the theoretical justification for why sequential RL works when sequential SFT would fail. The four arguments:

Argument 1: Policy-dependent data distribution. "In RL, the training data distribution is policy-dependent; the LLM generates its own experience. When a new objective or task is introduced, the LLM still explores across states, meaning old behaviors are continuously sampled if they remain useful or high-reward. This contrasts with supervised learning (e.g., SFT), where the samples in the previous domain disappear unless explicitly replayed." The key mechanism: if math reasoning remains reward-relevant during code training (because verbose, sloppy reasoning leads to buggy code), the policy will continue to generate math-like reasoning, and those samples will appear in the training distribution.

Argument 2: Reward optimization, not distribution matching. "RL optimizes expected cumulative reward, not exact targets for each input. As a result, the updates focus on improving long-term outcomes rather than explicitly fitting a new token-level distribution. Old knowledge that remains reward-relevant naturally persists." SFT minimizes per-token cross-entropy, which explicitly pushes the model toward a new data distribution. RL's policy gradient only cares about reward—if prior capabilities produce higher reward, they are reinforced, not overwritten.

Argument 3: Overlapping reward structures. "The reward structures of RLHF and RLVR overlap substantially across domains, such as math, code, reasoning, and instruction following, since they all aim to make outputs better, more accurate, and more aligned with human preferences or verification signals. For instance, reducing verbosity or hallucination generally benefits all domains." This means that a capability learned in one stage (e.g., concise reasoning from RLHF) continues to produce high rewards in later stages (e.g., efficient Code RL rollouts), so gradient updates do not push the model away from it.

Argument 4: Domain ordering and prompt separation. "We further minimize prompt overlap to the greatest extent possible, given that prompts across domains are generally already distinct. For instance, we remove all math and competitive programming–related prompts from the RLHF stage to reduce cross-domain interference. Furthermore, domain-wise RL is organized from more general domains (e.g., RLHF, instruction-following) to more specialized ones (e.g., math, code, SWE), preventing specialized capabilities from being overwritten by generic behaviors." The cascade order is deliberate: general skills first (they benefit all downstream stages), specialized skills last (they are least likely to interfere with prior capabilities).

The paper acknowledges a failure mode: "Catastrophic forgetting may still occur when the reward of a new domain sharply conflicts with that of a previous one (e.g., optimizing for concise responses versus detailed, step-by-step reasoning), particularly when prompts from different domains are semantically similar." The example given—RLHF's conciseness preference vs. Math RL's need for extended reasoning—is exactly the tension that the length-extension curriculum in Math RL is designed to navigate: the compression stage (24K) satisfies the conciseness preference, then the extension stages (32K, 40K) incrementally push back toward longer reasoning.

4. Key Insights and Innovations

Innovation 1: Sequential Domain-Wise RL Is Resistant to Catastrophic Forgetting — and This Changes the Default Training Paradigm

The paper's most consequential conceptual move is the argument that the field's default assumption — that diverse-domain RL must be done jointly to prevent forgetting — is wrong, and that this error stems from incorrectly importing intuitions from supervised learning. This is not an incremental optimization; it is a diagnosis of why prior approaches (DeepSeek-R1, Qwen3) accepted substantial engineering complexity as necessary, and a demonstration that the complexity can be eliminated without performance loss.

Before this work, the dominant paradigm for building general-purpose reasoning models was to blend heterogeneous prompts from all domains into joint RL stages. DeepSeek-R1 used a two-stage process: initial reasoning-oriented RL on math/code/science, then a second stage covering all domains with blended prompts. Qwen3 followed a similar pattern. The assumption — implicit but never challenged — was that if you trained sequentially, later stages would overwrite earlier capabilities, the way sequential SFT on disjoint datasets causes catastrophic forgetting.

The paper identifies four structural reasons why RL is different (Section 4.1.1), and each is a specific claim about RL dynamics, not a hand-wavy assertion:

Policy-dependent data distribution: In RL, the model generates its own experience. Old behaviors remain in the training distribution if they produce reward. In SFT, old-domain data simply disappears.
Reward optimization, not distribution matching: RL maximizes cumulative reward, not per-token cross-entropy. The model is not forced toward a new data distribution — it only needs to improve reward, and prior capabilities that remain reward-relevant are reinforced.
Overlapping reward structures across domains: Conciseness, accuracy, and reduced hallucination benefit math, code, alignment, and instruction-following simultaneously. The reward signals are not orthogonal.
Deliberate prompt separation and cascade ordering: Math and code prompts are excluded from RLHF; the cascade proceeds from general (RLHF) to specialized (SWE), so generic behaviors are learned first and not overwritten by later narrow training.

The empirical evidence for this claim is distributed across every results table in the paper. After Math RL, code benchmarks improve (Table 6: LCB v5 for 8B-Thinking goes from 69.0 to 71.2). After Code RL, math benchmarks show normal evaluation variance, not degradation (Table 7: AIME 2024 for 8B-Thinking drops 1.9 points but AIME 2025 rises 1.6). After SWE RL — the most specialized, final stage — all other benchmarks remain within ±1% of their Code RL levels (Table 8). The cascade does not just avoid catastrophic forgetting; it produces positive transfer from earlier stages to later ones, and no measurable forgetting from later stages to earlier capabilities.

This finding is significant beyond the specific Cascade RL recipe because it changes the default answer to "how should we train general-purpose reasoning models?" The paper's answer is: train sequentially, not jointly. This is a fundamental shift in training methodology justified by a specific mechanistic argument about RL dynamics, not a heuristic.

Innovation 2: RLHF as a Reasoning Enhancer, Not Just an Alignment Tax

The paper reframes the role of RLHF in the post-training pipeline through a specific empirical finding: RLHF applied before reasoning RL substantially improves reasoning benchmark performance, and the mechanism is verbosity reduction, not just preference alignment. This inverts the standard framing of RLHF as a capability-constraining alignment step.

The conventional wisdom — documented across the RLHF literature — is that alignment comes at a cost. Models optimized for human preference often lose raw capability: they become evasive, refuse edge cases, and produce safer but less creative or accurate responses. The term "alignment tax" captures this tradeoff. Even when RLHF is applied to reasoning models (as in DeepSeek-R1's final stage), it is typically positioned as a necessary constraint on top of already-achieved reasoning capability — you build the reasoner first, then align it.

The paper's finding is different and specific: RLHF applied first, before any reasoning-specific RL, improves AIME 2024 by 2.6 points (8B-Thinking: 83.6 → 86.4), GPQA Diamond by 3.0 points (64.2 → 67.3), and LiveCodeBench v5 by 10.7 points (59.6 → 70.3) — all despite the RLHF training data containing zero math or code prompts (Tables 2 and 4). The mechanism is traced to a concrete behavior change: the SFT models, trained on teacher-generated long-form reasoning, produce unnecessarily verbose chains of thought that waste tokens and sometimes prevent completion within the context budget. RLHF penalizes this (the reward model prefers concise, clear outputs), making the model more token-efficient.

The practical consequence is a reordering of the standard pipeline. Instead of reasoning RL first, then alignment, the optimal order is alignment first, then reasoning. The paper's ablation (Section 6.1, Figure 9) further shows that training RLHF on a mixture of thinking and non-thinking data (the "Half-Half" setting) yields better ArenaHard and better reasoning than thinking-only RLHF — cross-mode transfer from non-thinking training improves thinking-mode quality. This is a non-obvious finding that depends on the cascaded framework making each stage's contribution individually measurable.

Innovation 3: The Unified Model Gap Can Be Closed by Training Non-Thinking Mode in Early Stages

The paper identifies and resolves a specific failure mode that has plagued unified reasoning models: performance degradation in thinking mode when the model is trained on both thinking and non-thinking data. This is not a minor regression — it caused Qwen3 to revert from unified to separate thinking/instruct variants after release (Section 1). The paper's solution is conceptually simple but depends on an insight about when non-thinking training should occur in the cascade.

The key finding emerges from the IF-RL stage design (Section 4.4.2). The paper's unified 8B model applies RLHF in both thinking and non-thinking modes, then IF-RL only in non-thinking mode. This produces a model where thinking-mode instruction-following improves despite never being directly trained (IFEval in thinking mode: 85.3 for the unified model vs. 83.3 for the dedicated thinking model that did train IF-RL in thinking mode — Table 5). The final unified model closes the gap with the dedicated thinking model across reasoning benchmarks (Table 8: AIME 2024 89.5 vs. 88.8, LCB v5 74.3 vs. 74.5) while substantially outperforming it on instruction-following (IFEval 90.2 vs. 83.7, IFBench 40.8 vs. 41.4).

The conceptual move is: non-thinking training in early stages (RLHF) establishes robust instruction-following behavior without interfering with reasoning capability, because the reward signals for conciseness and helpfulness overlap with reasoning quality. Later stages (Math RL, Code RL) operate exclusively in thinking mode and build reasoning capability on top of this foundation without the interference that occurs when non-thinking and thinking training are interleaved throughout the pipeline.

This is significant as a diagnostic finding, not just a recipe. It suggests that the degradation observed in Qwen3 and other unified models is not an inherent capacity limitation of smaller models — the paper explicitly states it "challenges the assumption that LLMs, especially smaller ones, lack the capacity to learn effectively from both non-thinking and thinking data." Rather, the failure is in curriculum design: non-thinking training late in the pipeline interferes with reasoning; non-thinking training early in the pipeline enhances it.

Innovation 4: SWE RL Can Be Scaled Using Execution-Free Semantic Reward Models

The SWE RL stage introduces a methodological innovation with implications beyond the paper's specific results: replacing Docker-based execution verification with an LLM-as-judge reward model that never runs the generated code. This is not just an efficiency improvement — it enables training on datasets of arbitrary size by removing the bottleneck of managing Docker containers at scale.

Prior work on SWE RL (Jain et al., 2025; Luo et al., 2025a) relies on executing model-generated patches in Docker environments to obtain ground-truth correctness signals. This is accurate but severely limits scalability: "running and managing numerous Docker instances significantly limits scalability, constraining prior work to training datasets of around 10k unique instances" (Section 4.7.2). The consequence is that SWE RL has been restricted to small training sets, limiting the model's exposure to diverse bug-fixing patterns.

The paper's reward function (Section 4.7.2, Equation 2) replaces execution with a hybrid approach: perfect lexical matches get a reward of 1; unparseable patches get -1; no-change patches get 0; everything else gets a semantic similarity score from Kimi-Dev-72B, prompted with a yes/no question about whether the generated and golden patches achieve the same effect. The ablation (Table 12) shows this semantic judge outperforms purely lexical similarity (43.0% vs. 42.8% avg@4 with ground-truth localization) and does not require reward shaping — the LLM's probability signal is informative even at low similarity values.

The broader implication is that execution-free verifiers trained or prompted as LLM judges can substitute for ground-truth execution feedback in domains where execution is slow, expensive, or infrastructure-constrained. This is a specific instantiation of a more general idea — using LLMs as reward models for tasks where rule-based verification is unavailable — and the paper demonstrates it works at scale for a complex, repository-level software engineering task. The limitation (acknowledged by the authors) is that the approach still requires ground-truth patches for training, making it applicable to RL on existing bug-fix datasets but not to self-play or exploration-based SWE improvement.

Innovation 5: The Difficulty-Adaptive Length Extension Curriculum as a General RLVR Stabilizer

The three-stage length extension curriculum in Math RL (24K → 32K → 40K) is presented as a domain-specific technique, but the paper's description reveals it as a general strategy for stabilizing RLVR training when the initial policy exhibits pathological length behaviors. The insight is that each stage serves a distinct function — compression, stabilization, extension — and that ordering them correctly is essential.

The specific pathology addressed: SFT models trained on long-form teacher reasoning produce overlong chains of thought, with 15–20% incomplete ratios on AIME under a 32K budget. The standard response — simply train with a long budget from the start — fails because the model wastes tokens on verbosity rather than genuine reasoning. The alternative — train with a short budget and overlong penalty — also fails because assigning reward 0 to overlong responses "may excessively penalize long reasoning on difficult problems, causing a sharp performance drop" (Section 4.5.2).

The paper's solution is a carefully ordered sequence where overlong filtering is applied only in the compression stage, and then deliberately removed. Stage 1 (24K, with overlong filtering) forces the model to compress reasoning. Stage 2 (32K, without overlong filtering) allows natural expansion. Stage 3 (40K, without overlong filtering) pushes on hard problems specifically. The removal of overlong filtering after Stage 1 is the non-obvious design choice: it treats length not as a constraint to enforce but as a budget to fill, once the model has learned to use tokens efficiently.

This is significant beyond Math RL because it addresses a general challenge in RLVR: the initial policy's token allocation is often misaligned with what produces the best reward, and naive length penalties or budgets can destroy performance on hard problems. The staged approach — compress first, then extend — provides a template for applying RLVR to any domain where the initial policy exhibits verbosity or token inefficiency, which is common when fine-tuning on teacher-generated reasoning data. The paper does not claim this generality, but the structure of the solution (identify the pathology, stage responses to address it, remove constraints once the behavior is corrected) is domain-agnostic.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on a comprehensive suite of benchmarks spanning knowledge reasoning (MMLU, MMLU-Pro, GPQA-Diamond), alignment (ArenaHard, IFEval, IFBench), mathematical reasoning (AIME 2024, AIME 2025), competitive programming (LiveCodeBench v5/v6, LiveCodeBench Pro 25Q1/25Q2), software engineering (SWE-bench Verified), and tool use (BFCL V3). For competitive programming, the paper specifically evaluates only on problems released after the training data cutoff of August 2024 to prevent contamination. Additionally, the models are evaluated on the 2025 International Olympiad in Informatics (IOI), which imposes a limit of at most 50 submissions per problem with official judge feedback.

Base model(s). All experiments start from Qwen3-8B-Base and Qwen3-14B-Base (Yang et al., 2025a), chosen to enable transparent comparison and community reproducibility. The paper produces three model variants: a dedicated 8B-Thinking model, a unified 8B model (supporting both thinking and non-thinking modes), and a dedicated 14B-Thinking model. For the FLOPs-matched or teacher-student comparisons, DeepSeek-R1-0528 (671B) serves as the SFT teacher, with additional baselines including Qwen3-235B-A22B, Gemini-2.5-Pro-06-05, o4-mini, and various open-weight models at 7B–49B scale.

Metrics. The primary metric throughout is pass@1 accuracy, computed as the fraction of test instances where the model's final answer matches the ground truth. For small test sets (e.g., AIME with 30 problems, GPQA-Diamond with 198 problems), the paper reports avg@k — the average pass@1 computed over k independent generations per prompt, typically k=8 for coding and k=64 for AIME — to reduce evaluation variance. MMLU and MMLU-Pro use exact match (EM) with a single generation per question due to their large test set sizes (14,079 and 12,000+ questions). SWE-bench Verified reports avg@4 using the Agentless framework's resolve rate. For Codeforces contests, the paper reports Elo ratings computed by simulating contest participation with 8 independent submissions per problem, estimating expected penalties, ranking against human participants, and solving the implied Elo equation (Appendix E). The Elo computation involves the formula:

$m = \sum_{i=1}^n \frac{1}{1 + 10^{(R_{\text{model}} - R_i) / 400}}$

where $m$ is the model's estimated rank, $n$ is the number of human participants, and $R_i$ is the Elo rating of human contestant $i$ before each contest. Performance ratings are averaged over 51 Codeforces rounds.

Baselines. The paper compares against an extensive set of models. Proprietary front-tier models: o4-mini (high and medium reasoning effort), o3 (high), Gemini-2.5-Pro-06-05, Gemini-2.5-Flash-Thinking. Open-weight large models: DeepSeek-R1-0528 (671B), Qwen3-235B-A22B (thinking mode and dedicated thinking variant), Qwen3-Next-80B-A3B-Thinking. Open-weight models at comparable scale: Qwen3-8B, Qwen3-14B, Nemotron-Nano-9B-v2, Magistral-Medium-1.2-2509, OpenReasoning-Nemotron-32B, Llama-3.3-Nemotron-Super-49B-v1.5, Meta-CWM-32B, Klear-Reasoner-8B, AReaL-Boba-2-14B, AceReason-Nemotron variants, DeepSWE-32B (for SWE), and Ministral-3-14B-Reasoning-2512 (for SWE). For officially reported baseline results, the paper uses those directly; otherwise, it evaluates baselines with their recommended inference configurations or the same settings used for Nemotron-Cascade.

Generation budget / compute accounting. For reasoning benchmarks (AIME, LiveCodeBench, GPQA-Diamond), the thinking budget is set to 64K tokens maximum response length with YaRN scaling factor 2. For alignment benchmarks (ArenaHard, IFEval, IFBench), the maximum response length is 32K tokens. For SWE-bench, the thinking budget is 32K tokens with maximum input prompt lengths of 32K (8B) or 64K (14B) and YaRN scaling factors of 2 (8B) or 3 (14B). Sampling uses temperature 0.6 and top-p 0.95 for evaluation. All budgets are measured in tokens, not FLOPs — the paper focuses on generation budget rather than total computational cost, meaning the cost of the reward model, the RL training infrastructure, and the difficulty estimation are not included in the reported efficiency comparisons.

Cross-validation / statistical protocol. The paper does not report confidence intervals or formal statistical tests. For avg@k metrics, standard error decreases with k but is not quantified. The two-fold cross-validation used in earlier stages (for strategy selection across difficulty bins) is not applied to the main Cascade RL results — the evaluation is point-estimate based on the final checkpoint selected from each stage's training trajectory. Checkpoint selection criteria are not formally specified beyond monitoring training curves (e.g., Figure 4 for Math RL, Figure 6 for Code RL) and picking the best-performing step. This means the reported results may include an implicit winner's-curse effect from selecting the best checkpoint along a noisy training trajectory.

Main Quantitative Results

Overall Pipeline Performance (Table 1)

The final Nemotron-Cascade models achieve state-of-the-art performance for their size class across nearly all benchmarks. The 14B-Thinking model scores 85.1 on MMLU, 77.0 on MMLU-Pro, 69.6 on GPQA-Diamond, 89.7 on AIME 2024 (avg@64), 83.3 on AIME 2025, 77.5 on LiveCodeBench v5, 74.6 on LiveCodeBench v6, 68.9 on LiveCodeBench Pro 25Q2 Easy, 10.5 on LiveCodeBench Pro 25Q2 Medium, and 43.1 on SWE-bench Verified. The 8B unified model scores 83.7 on MMLU, 75.7 on MMLU-Pro, 66.5 on GPQA-Diamond, 89.5 on AIME 2024, 80.1 on AIME 2025, 74.3 on LCB v5, 71.1 on LCB v6, 65.7 on LCB Pro Easy, 6.4 on LCB Pro Medium, and 37.2 on SWE-bench. The improvements over the initial SFT models (indicated by ↑ numbers in Table 1) are dramatic: for the 8B unified model, LCB v5 improves by 15.1 points (59.2 → 74.3), IFEval improves by 19.4 points (70.8 → 90.2), and ArenaHard improves by 17.9 points (70.0 → 87.9). For the 14B-Thinking model, LCB v5 improves by 11.4 points (66.1 → 77.5), SWE-bench improves by 8.6 points (34.5 → 43.1), and IFBench improves by 17.4 points (24.3 → 41.7).

Competitive Programming Results (Table 9, Figure 1)

The 14B-Thinking model surpasses its 671B SFT teacher DeepSeek-R1-0528 across all competitive programming benchmarks. On LiveCodeBench v5: 77.5 vs. 74.8 (DeepSeek-R1-0528), 70.7 (Qwen3-235B-A22B thinking mode), 81.6 (Qwen3-235B-A22B-Thinking-2507), and 73.5 (Gemini-2.5-Pro-06-05). On LiveCodeBench v6: 74.6 vs. 73.3 (DeepSeek-R1-0528), 67.3 (Qwen3-235B-A22B thinking), 78.7 (Qwen3-235B-A22B-Thinking-2507), and 73.6 (Gemini-2.5-Pro-06-05). On LiveCodeBench Pro 25Q2: on Easy problems, 68.9 vs. 63.9 (DeepSeek-R1-0528) and 77.3 (Gemini-2.5-Pro); on Medium problems, 10.5 vs. 7.0 (DeepSeek-R1-0528) and 21.1 (Gemini-2.5-Pro). The Elo rating on 51 Codeforces rounds (2501–2507) is 1932 for 14B-Thinking and 1789 for 8B unified — the 8B model's Elo of 1789 places it at the 95.7th percentile, above OpenReasoning-Nemotron-32B (1766, 95.3rd) despite having 4× fewer parameters. The 14B model's Elo of 1932 beats Qwen3-235B-A22B-Thinking-2507 (1979) by a smaller margin than on LCB, suggesting Codeforces Elo captures different aspects of coding ability than pass@1 on curated benchmarks.

The 8B unified model achieves LCB v5 of 74.3, which is highly comparable to DeepSeek-R1-0528's 74.8 despite the 84× parameter gap. This specific comparison — an 8B student matching its 671B teacher — is one of the paper's strongest empirical claims, and it holds on LCB v5 but narrows on LCB v6 (71.1 vs. 73.3).

IOI 2025 Results (Figure 5)

The 14B-Thinking model achieves a total score of 343.37 on IOI 2025, corresponding to silver-medal performance, using the test-time scaling pipeline described in Section 5.1 with at most 1000 total generations (20 per round × 50 rounds) and no more than 50 official submissions per problem. On Problem 2 ("Triples"), the model scores 90.37 points, outperforming OpenAI's internal IOI-gold model (75.29) and DeepSeek-V3.2-Speciale (82 points, Liu et al., 2025a). Figure 5 (right) shows the progression over 27 rounds: the score rises from ~50 in early rounds to a final 90.37, with substantial jumps occurring when cross-subtask insights from solved subtasks are incorporated into later prompts. The left panel of Figure 5 shows the full problem set performance, though specific problem-by-problem scores are not tabled.

Per-Stage Progression (Tables 2, 4, 5, 6, 7, 8)

The Cascade RL pipeline's effects are tracked by evaluating the model after each RL stage. The progression for the 8B unified model on key benchmarks:

SFT → RLHF transformation (Table 2 → Table 4): The largest initial gains occur here, despite RLHF containing no domain-specific prompts. ArenaHard: 70.0 → 90.1 (+20.1). LCB v5: 59.2 → 70.2 (+11.0). AIME 2024: 83.6 → 86.1 (+2.5). IFEval: 70.8 → 50.1 (-20.7) — this is the one major regression, later recovered by IF-RL. GPQA Diamond: 63.5 → 66.8 (+3.3).
IF-RL recovery (Table 4 → Table 5): IFEval rebounds from 50.1 to 90.4 (+40.3). IFBench: 24.5 → 40.5 (+16.0). Reasoning benchmarks show minor fluctuations: AIME 2024 86.1 → 86.2 (+0.1), LCB v5 70.2 → 70.2 (unchanged), with MMLU-Pro dropping from 77.8 to 74.5 (-3.3) — the largest degradation attributed to "reduced model entropy" from IF-RL.
Math RL (Table 5 → Table 6): AIME 2024: 86.2 → 90.2 (+4.0). AIME 2025: 75.2 → 81.9 (+6.7). LCB v5: 70.2 → 70.6 (+0.4). SWE-bench: 28.3 → 30.6 (+2.3). MMLU-Pro partially recovers: 74.5 → 75.0 (+0.5).
Code RL (Table 6 → Table 7): LCB v5: 70.6 → 75.3 (+4.7). LCB v6: 67.4 → 71.5 (+4.1). GPQA Diamond: 65.7 → 67.4 (+1.7). AIME 2024 slightly drops: 90.2 → 89.1 (-1.1). AIME 2025: 81.9 → 80.5 (-1.4). These reversals on math benchmarks after Code RL are small and the paper attributes them to evaluation variance.
SWE RL (Table 7 → Table 8): SWE-bench: 31.6 → 37.2 (+5.6). All other benchmarks are within ±1% of their Code RL values. AIME 2024: 89.1 → 89.5 (+0.4). AIME 2025: 80.5 → 80.1 (-0.4). LCB v5: 75.3 → 74.3 (-1.0). The unified model's SWE-bench of 37.2 closes the gap with the dedicated 8B-Thinking model (38.5), which started from a higher SFT baseline (30.2 vs. 26.1).

Difficulty-Dependent Code Improvements (Figures 7, 8)

The topic-wise and difficulty-wise breakdown of LiveCodeBench v6 for the 8B unified model (Figures 7 and 8) reveals where Cascade RL's gains concentrate. Figure 8 shows accuracy and average reasoning token count on Easy, Medium, and Hard difficulty splits. Easy problems saturate above 99% after RLHF. Medium problems benefit from Math RL, with accuracy rising from approximately 68% (post-IF-RL) to approximately 72% (post-Math RL), then to approximately 78% (post-Code RL). Hard problems show the largest relative gains: from approximately 15% (post-SFT) to approximately 22% (post-RLHF), then to approximately 28% (post-Math RL), and finally to approximately 35% (post-Code RL). Token usage follows an inverse pattern: RLHF sharply reduces tokens (~60% reduction), IF-RL reduces further (~20% additional reduction), then Math RL and Code RL progressively increase token usage as the model learns to allocate more reasoning to harder problems.

Figure 7 breaks down by topic: Math-related topics (Math, Graph, Geometry) benefit substantially from Math RL; String and Data Structure benefit primarily from Code RL, with minimal gains during Math RL. This topic-level specificity provides evidence that Math RL and Code RL teach distinct, complementary skills rather than redundant general reasoning improvements.

SWE Test-Time Scaling (Figure 12)

Test-time scaling on SWE-bench Verified produces significant gains for both models. For Nemotron-Cascade-8B (Figure 12a): majority@2 is approximately 39.2%, plateauing around 40% at higher k. best@k consistently outperforms majority@k, reaching 43.6% at best@32. pass@k rises to 57.7% at k=32, indicating a 15.6-point gap between what the model can generate at least once and what the validation pipeline can reliably select. For Nemotron-Cascade-14B-Thinking (Figure 12b): majority@k starts at 50.7% at k=2 and reaches a similar plateau, while best@k rises to 53.8% at best@32. pass@k continues rising with k, suggesting that larger generation budgets would yield further best@k improvements even with the current validation pipeline, though the cost in inference compute scales linearly with k. The 14B model at best@32 (53.8%) is competitive with DeepSWE-32B (52.4%, Luo et al., 2025a), a 32B specialized model that uses execution-based verification, achieved with a general-purpose model less than half its size.

Ablation Studies and Robustness Checks

RLHF training mode for unified models (Figure 9): The "Half-Half" training strategy — splitting each RLHF batch evenly between thinking and non-thinking prompts — substantially outperforms both thinking-only and non-thinking-only RLHF. On ArenaHard (evaluated in thinking mode): Half-Half achieves approximately 90, thinking-only approximately 87, non-thinking-only approximately 82. On AIME25: Half-Half approximately 75%, thinking-only approximately 72.5%, non-thinking-only approximately 68%. On LiveCodeBench: Half-Half approximately 68.5%, thinking-only approximately 67%, non-thinking-only approximately 66%. The non-thinking-only mode particularly underperforms, suggesting that excluding thinking data during RLHF removes beneficial training signal even for tasks evaluated in thinking mode. This cross-mode transfer is a non-obvious finding that depends on the unified chat template making mode-switching explicit.

RM size for RLHF (Figure 10): Using AceReason-Nemotron-1.0-7B as the policy model, RLHF training with reward models ranging from 7B to 72B reveals that larger RMs produce better alignment and better downstream reasoning. ArenaHard with style control: 72B RM achieves approximately 47, 32B RM approximately 45, 14B RM approximately 44, 7B RM approximately 41 — but the 7B RM shows a much larger gap between standard ArenaHard (approximately 50) and style-controlled ArenaHard (approximately 41), indicating the 7B RM is vulnerable to reward hacking via response length. AIME25 accuracy: 72B RM approximately 56%, 7B RM approximately 53% — a 3% absolute difference. Code benchmarks are affected minimally (±1% across RM sizes). This ablation demonstrates that RM quality has downstream effects on reasoning performance, not just alignment metrics.

RLHF training stability techniques (Tables 10, 11): When using a small 7B RM, applying KL penalty (1e-3), sequence-level loss aggregation, and reward shaping extends stable training from 350 to 950 steps, and improves style-controlled ArenaHard from 43.05 to 45.76 (Table 10). However, when using the 72B RM (Table 11), these techniques are unnecessary: the bare configuration (KL=0, token-level loss, no reward shaping) achieves 91.04 on standard ArenaHard vs. 90.03 with all stability techniques, and 75.03 on AIME25 vs. 73.80. The paper's conclusion — that these techniques should be deployed only when training shows instability — is directly supported.

RM backbone choice (Table 3): Qwen3-14B in thinking mode dramatically underperforms as an RM backbone (RewardBench 46.38 overall) compared to Qwen2.5-Instruct-14B (93.22). Even with non-thinking mode enabled, Qwen3-14B (91.72) fails to match Qwen2.5-Instruct-14B (93.22). WorldPM-72B (94.08) is slightly outperformed by vanilla Qwen2.5-Instruct-72B (95.15). These results justify the choice of Qwen2.5-Instruct-72B as the RM backbone.

Code RL temperature (Figure 6): Temperature 1.0 yields higher final accuracy (approximately 71.5% at step 90) compared to temperature 0.8 (approximately 70%) and temperature 0.6 (approximately 68%), but the higher-temperature runs exhibit entropy instability — the average token entropy rises more sharply and shows greater variance toward the end of training. Temperature 0.6 produces the most stable entropy curve but plateaus at lower accuracy. This ablation reveals a fundamental tension in Code RL: high temperatures improve exploration and final performance but risk training instability.

SWE RL prompt length (Table 13): Training with maximum prompt lengths of 16K, 24K, 32K, and 40K shows that repair accuracy with ground-truth localization peaks at 32K (44.1% avg@4) but drops at 40K (42.8%). With top-4 localization (the realistic evaluation setting), 32K achieves 42.7% while 40K drops to 41.5%. The paper hypothesizes that Qwen3-14B-Base has limited long-context capability around 32K, making 40K prompts introduce noise into the RL training signal rather than providing useful additional context.

SWE RL reward function (Table 12): Semantic similarity (Kimi-Dev-72B judge) as the continuous reward component outperforms lexical similarity (Unidiff). With ground-truth localization: semantic (cond. 3) achieves 43.0% avg@4 vs. lexical (cond. 1) at 42.6%. With top-4 localization: semantic (cond. 3) achieves 42.3% vs. lexical (cond. 1) at 41.2%. Reward shaping (setting rewards below 0.5 to 0) helps lexical similarity (cond. 2: 42.8% vs. cond. 1: 42.6% with ground-truth) but does not help semantic similarity (cond. 4: 42.9% vs. cond. 3: 43.0%), indicating that the LLM judge provides meaningful signal even at low similarity values where lexical overlap is unreliable.

SWE localization approach (Figure 11): The retrieval-based method (NV-Embed-Code) consistently outperforms the generation-based method across all top-k cutoffs for both 8B and 14B models. At top-1: retrieval achieves approximately 72% recall for 8B vs. approximately 60% for generation (single rollout), improving to approximately 66% with 16-rollout aggregation. Combining generation (16 rollouts) and retrieval via reciprocal rank fusion provides marginal additional improvements, most notably at cutoffs below top-5. The generation-based approach shows consistent gains from aggregating multiple rollouts, particularly at higher ranks, indicating that rollout diversity in localization complements the accuracy improvements from fusion.

IF-RL mode strategy (not explicitly ablated, but implied by Table 5): The unified 8B model achieves IFEval 90.4 after IF-RL applied only in non-thinking mode (Table 5), compared to 8B-Thinking's IFEval 83.3 after IF-RL applied in thinking mode with the combined RM+IF reward function. Meanwhile, ArenaHard degrades less for the unified model (88.0 vs. 90.1 pre-IF-RL, -2.1) than for the dedicated thinking model (86.3 vs. 89.9 pre-IF-RL, -3.6). This pair of comparisons — though not a controlled ablation — suggests that IF-RL in non-thinking mode is both more effective at improving IF capabilities and less damaging to alignment than IF-RL in thinking mode.

Critical Assessment

Claim: Cascade RL achieves 4× better efficiency than joint training, or equivalently that sequential training matches joint training while simplifying infrastructure. The paper does not actually run a joint-training baseline. The comparison is implicit: Nemotron-Cascade models outperform models that used joint training (DeepSeek-R1-0528, Qwen3-235B-A22B) on specific benchmarks, but these comparisons confound many variables beyond training paradigm — base model architecture, pretraining data, total compute budget, and SFT data quality all differ. The claim that Cascade RL "reduces engineering complexity" relative to joint training is an argument from principle (Section 4.1.1) supported by the observation that domain-specific hyperparameters can be tuned independently, but the paper provides no head-to-head comparison of Cascade RL vs. joint training on the same base model with the same total compute budget. This is the single most important missing experiment. The paper would have been substantially strengthened by training a Qwen3-8B-Base model with all domains blended jointly in a single RL stage (matching the total steps of Cascade RL) and comparing performance, infrastructure complexity, and training wall-clock time.

Claim: Subsequent RL stages rarely degrade performance attained in earlier domains. This is well-supported by the per-stage tables (Tables 4–8), which show fluctuations within ±3% on most benchmarks after the RLHF → IF-RL transition, and within ±1.5% for subsequent transitions. The largest degradation is IFEval dropping 20.7 points during RLHF (Table 4), but this is explicitly a domain not targeted by RLHF and is recovered by the subsequent IF-RL stage designed for it. The claim holds for the specific cascade order tested (RLHF → IF-RL → Math → Code → SWE). The paper does not ablate cascade order to test whether, for example, Math RL before RLHF would cause forgetting of math capability during RLHF. The theoretical argument (Section 4.1.1) suggests that general-to-specific ordering matters, and that reversing the order could cause forgetting — but this is untested.

Claim: RLHF boosts reasoning ability far beyond preference optimization, and this is a general phenomenon. Supported within the paper's specific setup: RLHF on non-math/non-code data improves AIME 2024 by 2.6 points and LCB v5 by 10.7 points (Table 4 vs. Table 2). The mechanism — verbosity reduction making the model more token-efficient — is supported by the observation that RLHF uses a short 12K response budget and that RLHF substantially reduces reasoning token counts (Figure 8). However, the claim's generality is untested: this effect depends on the SFT model being overly verbose to begin with (because it was trained on long-form teacher outputs from DeepSeek-R1). An SFT model trained on more concise reasoning data might not benefit from RLHF's verbosity reduction. The paper acknowledges this dependence: "RLHF substantially improves the quality of generated responses by reducing verbosity and repetition" — if verbosity is not the bottleneck, RLHF might not provide reasoning gains.

Claim: The unified 8B model closes the reasoning gap with the dedicated 8B-Thinking model. The final comparison (Table 8) shows 8B unified vs. 8B-Thinking: MMLU 83.7 vs. 84.0, MMLU-Pro 75.7 vs. 75.5, GPQA 66.5 vs. 66.7, AIME 2024 89.5 vs. 88.8, AIME 2025 80.1 vs. 81.4, LCB v5 74.3 vs. 74.5, LCB v6 71.1 vs. 71.4, SWE 37.2 vs. 38.5. The largest differences are 1.3 points on AIME 2025 (in favor of 8B-Thinking) and 1.3 points on SWE-bench (in favor of 8B-Thinking), offset by 0.7 points on AIME 2024 (in favor of unified). These differences are within typical evaluation variance for test sets of 30 (AIME) or 500 (SWE) instances. The unified model substantially outperforms 8B-Thinking on IFEval (90.2 vs. 83.7) and IFBench (40.8 vs. 41.4). The gap is effectively closed for reasoning and substantially reversed for instruction-following. The limitation: this comparison is between two models trained with slightly different recipes (the dedicated thinking model uses combined RM+IF reward for IF-RL in thinking mode, while the unified model uses IF-RL in non-thinking mode only). It is not a controlled ablation of the unified vs. dedicated design choice — other training differences exist.

Claim: The 14B model outperforms its 671B SFT teacher DeepSeek-R1-0528 on LiveCodeBench. This is the paper's strongest empirical result. On LCB v5: 77.5 vs. 74.8 (+2.7). On LCB v6: 74.6 vs. 73.3 (+1.3). On LCB Pro Easy: 68.9 vs. 63.9 (+5.0). On LCB Pro Medium: 10.5 vs. 7.0 (+3.5). These are non-trivial margins on test sets of 279–454 problems. However, DeepSeek-R1-0528 is not necessarily compute-matched to the Cascade RL process — its total RL compute budget, number of RL steps, and training data volume are unknown and likely larger. The comparison demonstrates that a smaller model with intensive RL can surpass a larger model with its own (undisclosed) training recipe on specific benchmarks, but it does not isolate Cascade RL as the causal factor. Moreover, DeepSeek-R1-0528 is an older checkpoint (May 2025) and may not represent the frontier of what joint training can achieve.

Missing experiments and genuine weaknesses:

No joint-training baseline on the same base model. This is the largest gap. Without it, the efficiency claims about Cascade RL vs. joint training are unquantified.
No cascade-order ablation. The paper argues that general-to-specific ordering matters (Section 4.1.1) but never tests, for example, Math RL before RLHF. This leaves open whether the benefits are from sequential training per se or from this specific sequence.
Single base model family (Qwen3). All experiments use Qwen3-8B-Base and Qwen3-14B-Base. Whether Cascade RL generalizes to other architectures (e.g., Llama, DeepSeek-V3) is untested. The paper's claim to provide an "open recipe" implies generality, but the recipe may depend on Qwen3-specific properties (base model verbosity, initial reasoning capability, long-context capacity).
Evaluation on limited benchmarks. The paper emphasizes general-purpose reasoning but evaluates primarily on math and code. Creative writing, multi-turn dialogue, safety, and multilingual capabilities are not systematically evaluated despite the SFT data covering these domains. ArenaHard serves as a proxy for general alignment quality, but it is a single benchmark with known biases.
Difficulty estimation cost is not quantified. Unlike the earlier example paper, Cascade RL does not require per-prompt difficulty estimation. However, the training process requires training a 72B reward model, running five sequential RL stages, and curating domain-specific datasets for each stage. The total compute cost of this pipeline is not reported, making it impossible to assess whether the final model performance justifies the training investment compared to simpler approaches (e.g., SFT on more data, or a single joint RL stage).
Test sets are small for some high-variance benchmarks. AIME 2024/2025 each contain only 30 problems. The paper reports pass@1 averaged over 64 generations (avg@64), which reduces variance but does not eliminate it — the standard error on a 30-question test set is approximately 2.8 percentage points at 50% accuracy with avg@64. Differences of 1–2 points between stages may be within noise. The paper does not report confidence intervals, making these small fluctuations difficult to interpret.
SWE-bench evaluation uses ground-truth localization for some ablations. Tables 12 and 13 report repair accuracy "with ground-truth localization" — meaning the model is given the correct buggy files rather than needing to find them. This is a simplified setting that does not reflect end-to-end SWE performance. The "top-4 localization" setting is more realistic, and the gap between the two settings (e.g., 43.0% vs. 42.3% in Table 12, cond. 3) shows that localization errors cost approximately 1 percentage point, which is modest but non-zero.
Reward model evaluation relies on RewardBench as a proxy. The paper acknowledges (Section 4.2.2) that "RewardBench can be an imperfect proxy for identifying the optimal reward model for RLHF, and the RLHF process itself introduces additional variance." The finding that the highest-RewardBench RM does not necessarily produce the best policy is noted but not deeply investigated — the paper selects the 72B RM primarily based on RewardBench and the ArenaHard style-control ablation.

Conditional nature of claims:

The central claim — that Cascade RL is an effective and efficient paradigm for building general-purpose reasoning models — holds for Qwen3-8B/14B base models trained with the specific sequence RLHF → IF-RL → Math RL → Code RL → SWE RL, evaluated on the specific benchmarks in Table 1. The paper provides strong evidence that sequential training does not cause catastrophic forgetting under these conditions, and that the approach yields models competitive with or exceeding much larger models on math and code. The claim that Cascade RL generalizes to other base models, other domain orderings, or other task distributions (e.g., tasks requiring more factual knowledge than reasoning) is plausible but unverified. The claim that Cascade RL is superior to joint training — as opposed to merely effective — is not directly tested and remains a hypothesis supported by engineering intuition rather than experimental evidence.

6. Limitations and Trade-offs

No Head-to-Head Comparison Against Joint RL Training on the Same Base Model

The paper's central architectural claim is that Cascade RL "reduces engineering complexity" compared to blending heterogeneous prompts into joint RL stages, while matching or exceeding performance. The introduction states:

"Such variability complicates the RL infrastructure, slows training, and makes training curriculum (e.g., response length extension) and hyperparameter selection more challenging."

Section 4.1.1 provides a theoretical argument for why sequential RL resists catastrophic forgetting. Every results table shows that benchmarks from earlier stages do not degrade after later stages. However, the paper never trains a joint-RL baseline on the same Qwen3 base model with the same total compute budget. The comparisons are always against external models—DeepSeek-R1-0528, Qwen3-235B-A22B—that differ in base architecture, pretraining data, SFT recipe, and total RL compute. These comparisons demonstrate that Cascade RL produces strong models, but they do not isolate the training paradigm as the causal factor.

The consequence is that the paper's strongest practical claim—that practitioners should prefer sequential over joint RL—rests on an engineering argument from first principles rather than controlled experimental evidence. A joint-training run might match or exceed Cascade RL performance on the same base model while being simpler in some respects (fewer checkpoint transitions, no need to curate disjoint prompt sets). The paper cannot rule this out. The claim that domain-wise RL enables "hyperparameters and training curriculum [to] be tailored to each specific domain for optimal performance" is true of Cascade RL, but joint training could potentially achieve similar domain-specific tuning through techniques like per-domain loss weighting or gradient accumulation—approaches the paper does not compare against.

The paper provides no evidence that would let a practitioner estimate the performance gap (if any) between Cascade RL and joint training. The theoretical argument in Section 4.1.1 describes mechanisms by which forgetting could be avoided, but does not quantify the risk of forgetting under joint training. The ablation in Section 5.2 (Figure 6) shows that Code RL is sensitive to temperature, which the paper uses to argue that joint training would force a suboptimal compromise—but this is a suggestive finding, not a comparative one. A joint-training run that allowed per-domain temperature scheduling would test whether this compromise actually matters in practice.

Mitigation status: The paper does not acknowledge this as a limitation. Section 8.1 positions Cascade RL against prior work by stating that joint training "complicates the RL infrastructure, training curriculum, and hyperparameter tuning, ultimately leading to suboptimal performance"—but "suboptimal performance" is not demonstrated relative to a joint-training baseline on the same model, only inferred from comparisons against external models. A fair joint-training baseline run on Qwen3-8B-Base would substantially strengthen the paper's central claim.

The RLHF-Improves-Reasoning Effect Depends Critically on SFT Model Verbosity

The paper reports that RLHF—applied first in the cascade, with zero math or code prompts—improves AIME 2024 by 2.6 points (8B-Thinking: 83.6 → 86.4, Table 4 vs. Table 2) and LiveCodeBench v5 by 10.7 points (59.6 → 70.3). This is presented as a finding about RLHF's general utility: "RLHF substantially improves overall response quality (e.g., reduces verbosity), thereby enhancing reasoning performance" (Section 1). The mechanism identified in Section 4.3.2 is:

"RLHF tends to reduce repetition and verbosity, thereby compressing the number of thinking tokens for simpler questions. This, in turn, enhances reasoning efficiency and training stability in the subsequent Math RL and Code RL stages."

The assumption embedded here is that the SFT model is pathologically verbose—a direct consequence of training on long-form Chain-of-Thought outputs from DeepSeek-R1 and DeepSeek-R1-0528 teachers. These teachers produce reasoning chains that the paper itself describes (in the context of Math RL) as "overlong reasoning chains, leading to incomplete ratios of 15–20% on the AIME benchmark under a 32K token budget" (Section 4.5.2). RLHF fixes this by penalizing verbosity via the reward model's preference for concise, clear outputs.

The consequence is that the RLHF-as-reasoning-enhancer result is not a general property of RLHF—it is a targeted correction of a specific SFT artifact. If the SFT model were trained on more concise reasoning data (e.g., from a teacher that produces efficient chains of thought, or with a response-length penalty during SFT), RLHF might provide minimal or zero reasoning gains. The paper does not test this. A practitioner attempting to replicate Cascade RL with a different teacher model or SFT recipe might find that RLHF provides negligible reasoning improvement, altering the optimal cascade order or making RLHF an unnecessary stage for reasoning purposes. The paper's strong claim (Abstract: "RLHF for alignment, when used as a pre-step, boosts the model's reasoning ability far beyond mere preference optimization") would be misleading in such a context.

Figure 8 provides indirect evidence for the dependency: it shows that reasoning token counts drop sharply (by ~60%) after RLHF, with accuracy simultaneously rising—consistent with the verbosity-correction mechanism. But the figure also shows that Easy problems are already above 99% accuracy after SFT, so the accuracy gains from RLHF's verbosity reduction are concentrated on Medium and Hard problems where SFT's verbosity was causing incomplete generations or wasted tokens. This pattern is exactly what you would expect if the gains come from fixing an artifact of SFT data, not from RLHF teaching fundamentally new reasoning strategies.

Mitigation status: The paper does not discuss this dependency. Section 4.5.2 states that "initialization from RLHF-trained models is highly beneficial because it provides a much stronger initial math reasoning capability than SFT checkpoints" and that "response quality is substantially improved, and reasoning becomes more token-efficient after RLHF." The causal link—SFT verbosity → RLHF fixes it → downstream gains—is implicit in the mechanism description but never qualified as a condition that might not hold under different SFT recipes. A simple ablation comparing RLHF's effect on SFT models trained with different teacher lengths or with explicit length penalties would clarify the generality of this finding.

Total Training Compute Cost Is Not Reported or Accounted For

The paper evaluates model performance as a function of inference budget (64K-token thinking budget, avg@k evaluation protocol) but does not report the total computational cost of producing those models. The Cascade RL pipeline involves: (1) two-stage SFT on 2.77M math samples, 1.42M code samples, 2.8M general-domain samples, and 127K SWE repair samples, totaling millions of teacher-generated responses; (2) training a 72B reward model on 82K preference pairs; (3) five sequential RL stages—RLHF (800–900 steps), IF-RL (two stages, ~1100–3100 total steps), Math RL (three stages, ~500 steps), Code RL (64–90 steps), SWE RL (60–120 steps)—using GRPO with 8–16 rollouts per prompt at batch sizes of 128–256; (4) training a separate ORM for revision-model outputs (mentioned for the earlier AceReason work but not for Cascade RL specifically). Each RL step requires generating G rollouts from the current policy then performing one gradient update, making the generation cost alone substantial (e.g., Math RL at 500 steps × batch 128 × 8 rollouts = 512K complete generations, each potentially 40K tokens long).

The consequence is that a practitioner cannot assess whether Cascade RL's performance gains justify its training cost compared to alternatives. Could additional SFT data, a larger SFT model, or a single longer joint-RL stage achieve comparable performance with less total compute? The paper's headline result—14B surpassing 671B teacher—is impressive in terms of inference-time efficiency (a small model outperforms a large one), but the total training FLOPs (SFT + RM training + 5 RL stages) might approach or exceed those of training the larger model. The paper's claim to provide a cost-effective alternative to "blending heterogeneous prompts" (Section 1) is an argument about engineering simplicity, not about total compute efficiency. The two could diverge: sequential training might be simpler to orchestrate but more expensive in total FLOPs if each stage requires substantial training to overcome the lack of joint optimization.

The paper also does not report the cost of the 72B reward model inference during RLHF. Each RLHF training step requires scoring every generated response with the RM—a forward pass through a 72B model. At 800 steps × batch 256 × rollout 8 = 1.6M RM inferences per RLHF run, this is non-trivial. Similarly, the SWE RL stage uses Kimi-Dev-72B as a reward model (Section 4.7.2), adding further external model inference cost. The asynchronous reward computation for Code RL (Section 4.6.2) reduces wall-clock time from 1172.4 seconds to 416.2 seconds per batch but does not reduce total FLOPs—the verification is simply parallelized, not eliminated. None of these costs are aggregated or compared against baselines.

Mitigation status: The paper does not address this limitation. The term "efficiency" appears only in the context of inference-time token usage (RLHF "enhances reasoning efficiency," Math RL "improves token efficiency") and engineering complexity ("departing from conventional approaches that blend heterogeneous prompts... reducing engineering complexity"). The paper never claims Cascade RL is more compute-efficient in total FLOPs than alternatives, but the absence of any cost accounting makes it impossible for a reader to assess whether the sequential approach is practical at scale or whether simpler alternatives (e.g., SFT on more data with a stronger teacher, followed by a single domain-blended RL stage) would achieve similar results with less total investment.

Hardest Problems Remain Essentially Unsolved Across All Methods

The paper acknowledges in multiple places that its approach provides negligible gains on the hardest problems in each domain. Section 4.5.2, describing Math RL:

"After 32K training, model accuracy on easy and medium problems nearly saturates (99% and 85% respectively) on AIME24/25, but hard problems remain challenging, with accuracy plateauing below 30%."

The 40K stage "explicitly incentivizes the model to leverage more tokens" and pushes hard-problem accuracy "from 30 to 40%," but this is the ceiling—4 out of 10 hard AIME problems remain unsolved. For competitive programming, the pattern is starker. On LiveCodeBench Pro Medium problems (the harder half of Codeforces-level contest problems), the 14B-Thinking model achieves only 10.5% (Table 9)—substantially trailing Gemini-2.5-Pro (21.1%) and o4-mini-high (29.8%). The Elo analysis (Appendix E, Tables 21–22) reveals extreme variance: the 14B model achieves estimated Elo above 2600 on some Codeforces rounds and below 1000 on others. The paper comments:

"We find inconsistent behavior on coding problem solving: while the model is sometimes able to solve very difficult problems, it can also become stuck on relatively easy ones, even within the same contest."

On SWE-bench Verified, even with test-time scaling to best@32, the 14B-Thinking model achieves 53.8% pass@1 (Figure 12b)—meaning nearly half of real-world GitHub issues remain unresolved. The specialized DeepSWE-32B achieves a comparable 52.4%, but neither figure represents a solved problem for deployment in automated software engineering pipelines.

The consequence is that Cascade RL, like the test-time compute scaling analyzed in the earlier example paper, amplifies existing capability but does not create it. If the base model (after SFT) cannot produce a correct reasoning chain for a problem—even occasionally—no amount of sequential RL will help, because RL only reinforces behaviors the policy can already sample. The Math RL dynamic filtering (Section 4.5.2) explicitly excludes problems with 0% accuracy as providing "no useful policy gradient signal." This is correct for training efficiency, but it means the model never improves on problems that are fundamentally outside its reach. The paper does not claim otherwise—Section 8.1 states that RLVR "has achieved remarkable success in developing frontier reasoning models"—but the "remarkable success" is concentrated in the regime where the base model already has non-trivial capability.

The topic-level analysis (Figure 7) provides additional evidence: Code RL provides strong gains on String and Data Structure problems but minimal gains on topics that the SFT model already handles well (Math, Graph, Geometry after Math RL). The gains are largest where the model has room to improve but non-zero starting capability—exactly the pattern observed in the test-time compute scaling paper. The practical implication is the same: Cascade RL is a powerful amplification mechanism but not a substitute for better pretraining or SFT data on genuinely challenging problems.

Mitigation status: The paper is transparent about this limitation in the per-domain analyses but does not elevate it to a general property of the Cascade RL paradigm. Section 8.1 notes that prior work used "joint training" and "diverse prompts" in an attempt to handle cross-domain heterogeneity, implying that Cascade RL handles this better—but neither approach solves the fundamental capability ceiling. The paper does not discuss whether specific domain orderings, repeated cycling through domains, or incorporating unsolved problems via exploration bonuses could break through the ceiling, leaving this as entirely future work.

Single Base Model Family and Limited Domain Coverage Constrain Generalizability Claims

All experiments use Qwen3-8B-Base and Qwen3-14B-Base as starting points. The paper states in Section 1:

"We focus on developing an open post-training recipe, using the pretrained Qwen3-8B-Base and Qwen3-14B-Base as starting points to support transparent comparison and facilitate knowledge sharing within the community."

This is a reasonable scope for a single paper, but it limits the generality of the Cascade RL findings in specific ways that affect practitioners:

Qwen3-specific properties may be load-bearing. The Qwen3 base models were pretrained with a 32K context window. The SWE RL ablation (Table 13) shows that extending prompt length beyond 32K causes performance degradation—"the pretrained Qwen3-14B-Base has limited long-context capability at 32K"—but this is a property of the base model's pretraining, not of Cascade RL. A base model with native 128K context (e.g., Command R, Gemini, Llama 4) might not need the multi-stage context extension curriculum, or might benefit from even longer SWE RL prompts. The optimal cascade ordering might differ if the base model already exhibits efficient reasoning without RLHF's verbosity reduction.
The RLHF-improves-reasoning effect may not transfer. As discussed above, this effect depends on SFT verbosity. SFT verbosity depends on the teacher model (DeepSeek-R1 family) and the base model's tendency to imitate teacher length. A Llama-base model fine-tuned on the same data might exhibit different verbosity patterns, altering whether RLHF-first is optimal.
Domain ordering was tested in exactly one configuration. The cascade order (RLHF → IF-RL → Math → Code → SWE) is justified by theoretical arguments in Section 4.1.1—general-to-specific, prompt separation, overlapping reward structures—but no alternative ordering is ablated. Would IF-RL before RLHF produce a different result? Would Code RL before Math RL (the reverse of AceReason-Nemotron's finding that Math → Code is optimal) degrade math performance? The paper cannot answer these questions. The argument that "subsequent domain-wise RLVR stages rarely degrade the benchmark performance attained in earlier domains" (Section 1) is supported for this specific ordering but not for Cascade RL as a domain-ordering-agnostic paradigm.
Evaluation skews toward math and code. The paper's headline benchmarks (AIME, LiveCodeBench, SWE-bench) all measure mathematical reasoning, algorithmic problem-solving, or software engineering. The SFT data covers creative writing, role-playing, safety, and multi-turn dialogue (Section 3.2.1), but these capabilities are evaluated only via ArenaHard (a single 500-prompt benchmark using LLM-as-judge) and IFEval/IFBench (instruction-following constraint satisfaction). There is no evaluation of whether Cascade RL preserves or degrades creative writing quality, factual accuracy in open-ended generation, safety alignment, or multilingual capability. The claim to build "general-purpose reasoning models" (title, Section 1) is supported for reasoning benchmarks but not for the full spectrum of general-purpose LLM capabilities.

Mitigation status: The paper acknowledges the Qwen3-specific scope in Section 1 as a deliberate choice for reproducibility. It does not claim that Cascade RL generalizes to other base models—this is left implicit. The domain coverage limitation is partially acknowledged by the benchmarks chosen (the paper evaluates what it can measure with automated metrics) but the gap between "general-purpose reasoning" and the evaluated capabilities is not discussed. Future work on applying Cascade RL to other base model families and evaluating on a wider range of capabilities is implied but not explicitly called out.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the default assumption about how post-training RL should be structured for general-purpose reasoning models. Before this work, the dominant paradigm—exemplified by DeepSeek-R1, Qwen3, and their derivatives—was to blend heterogeneous prompts from math, code, science, alignment, and instruction-following into joint RL stages. The engineering complexity was accepted as necessary, justified by the intuition (imported from supervised learning) that sequential training on disjoint distributions would cause catastrophic forgetting of earlier capabilities. Cascade RL provides systematic empirical evidence that this intuition does not transfer to the RL setting, and that the assumed necessity of joint training was an untested constraint on pipeline design.

The magnitude of this shift is more diagnostic than paradigmatic. The paper does not propose a fundamentally new learning algorithm—it uses GRPO, an established method. It does not introduce a new model architecture or a new reward mechanism. What it provides is a rebuttal to a specific, widely-held assumption: that training diverse capabilities sequentially will overwrite them. The evidence is distributed across every results table (Tables 4–8), showing that benchmarks from earlier stages remain within ±1–3% after later stages, and in several cases improve. The mechanism analysis in Section 4.1.1—policy-dependent data distribution, reward optimization rather than distribution matching, overlapping reward structures across domains—provides the conceptual rationale for why RL behaves differently from SFT in this regard.

This reframing changes which research directions become attractive and which become less so:

More attractive: Domain-wise RL with per-domain hyperparameter optimization becomes the natural next step, not a compromise. The paper demonstrates that Code RL requires temperature 1.0 for optimal exploration (Figure 6), RLHF requires a tight 12K response budget to suppress verbosity (Section 4.3.2), and Math RL requires a staged 24K → 32K → 40K length extension curriculum (Section 4.5.2). Joint training would force a single temperature setting and a single response-length schedule onto all domains—a compromise that Cascade RL avoids entirely. The finding that these per-domain optimizations matter (the temperature ablation shows a ~3% gap between temperature 0.6 and 1.0 for Code RL; the Math RL curriculum is essential for recovering from SFT verbosity) suggests that joint training may be leaving substantial performance on the table, not just adding engineering complexity.

Less attractive: The paper makes elaborate joint-training infrastructure design—the kind needed to handle heterogeneous verification latencies, per-domain gradient accumulation, and mixed reward signals in a single training loop—appear unnecessary rather than impressive. If sequential RL works without catastrophic forgetting, the primary motivation for building infrastructure that can handle code execution, symbolic math verification, and reward model scoring in the same batch evaporates. Research effort shifts from "how do we make joint training work despite heterogeneity" to "what is the optimal sequence and per-stage configuration for domain-wise RL." This is a simpler research program with clearer ablations.

Reconciling prior contradictions: The paper provides a potential explanation for a tension in the RLHF literature. Some works find RLHF degrades reasoning capability (the "alignment tax" framing), while others find it can be neutral or positive. The Cascade RL results suggest the outcome depends critically on when RLHF is applied and what condition the model is in. When applied to a pathologically verbose SFT checkpoint trained on long-form teacher outputs, RLHF improves reasoning by compressing wasted tokens (the mechanism in Section 4.3.2). When applied to a model that has already undergone reasoning RL and learned efficient chains of thought, RLHF's verbosity penalty might suppress necessary reasoning steps, causing degradation. Prior studies may have been testing RLHF at different points in the training pipeline on models with different verbosity profiles, producing contradictory results that Cascade RL's staged analysis helps reconcile.

A new diagnostic capability: The paper's per-stage evaluation tables (Tables 2, 4, 5, 6, 7, 8) provide a template for debugging post-training pipelines. When a model's performance on a benchmark degrades, the staged approach makes it possible to identify which stage caused the regression, rather than treating the entire RL process as a black box. The paper uses this diagnostically throughout: identifying that RLHF causes IFEval to drop (Table 4), that IF-RL recovers it (Table 5), that IF-RL temporarily reduces model entropy and slightly degrades reasoning (Table 5), and that Math RL and Code RL restore and exceed the pre-IF-RL reasoning levels (Tables 6, 7). This kind of stage-level attribution is impossible in joint training, where all domains are updated simultaneously and capability changes cannot be traced to specific training decisions.

The execution-free SWE reward model finding: This is a methodological contribution that may influence research beyond the specific Cascade RL context. The demonstration that Kimi-Dev-72B semantic similarity scores can substitute for Docker-based execution verification (Table 12: semantic similarity achieves 43.0% vs. 42.6% for lexical similarity with ground-truth localization, and the gap widens with retrieval-based localization) opens the door to scaling SWE RL to much larger datasets. The paper's argument—that Docker execution limits prior work to ~10K training instances—identifies a specific bottleneck, and the execution-free reward provides a concrete path around it. This is not a solved problem (the approach still requires ground-truth patches, so it does not enable self-play or exploration), but it changes the scaling properties of SWE RL from "constrained by Docker infrastructure" to "constrained by ground-truth patch availability."

Follow-Up Research This Work Enables

Directly compare Cascade RL to joint training on the same base model with the same total compute budget. This is the single most important missing experiment. The paper argues that Cascade RL is simpler and more effective than joint training, but the evidence is always against external models (DeepSeek-R1-0528, Qwen3) with different architectures, pretraining data, and training recipes. A controlled experiment would take Qwen3-8B-Base, replicate the SFT stage identically, then train two variants: one with the Cascade RL sequence (RLHF → IF-RL → Math → Code → SWE, totaling ~2000–3500 steps depending on model variant), and one with a joint RL stage that blends all domain prompts together, using the same total number of gradient steps and the same per-domain data volume. The comparison would measure final benchmark performance, training wall-clock time, and infrastructure complexity (lines of configuration code, number of manual interventions, failure recovery difficulty). This experiment would directly quantify whether Cascade RL's infrastructure simplification comes at any performance cost, or whether it is strictly dominant. The paper provides all necessary training details (hyperparameters, data blends, step counts) to make this replication feasible.

Ablate cascade ordering to identify which sequences cause forgetting and which produce positive transfer. The paper's theoretical argument (Section 4.1.1) claims that general-to-specific ordering matters because specialized capabilities are less likely to overwrite general ones than vice versa, and that overlapping reward structures prevent interference. But this is untested. Specific ordering ablations would include: (a) Math RL before RLHF—does RLHF's verbosity penalty degrade the extended reasoning chains learned during Math RL, or does the policy retain efficient math reasoning? (b) Code RL before Math RL—the reverse of AceReason-Nemotron's finding that Math → Code is optimal; does this degrade math performance or improve code performance compared to the original order? (c) IF-RL before RLHF—does instruction-following training degrade alignment quality more than the current RLHF-first order? (d) Cycling: RLHF → Math → RLHF → Code → RLHF, testing whether revisiting earlier domains provides additional gains or merely wastes compute. Each ablation would use the same SFT checkpoint, the same per-stage hyperparameters, and the same total steps per domain, with the only variable being ordering. The results would transform Cascade RL from a specific recipe into a general principle with known boundary conditions.

Test whether the RLHF-improves-reasoning effect generalizes to SFT models with different verbosity profiles. The paper's mechanism—RLHF reduces verbosity, which makes reasoning more token-efficient and improves accuracy within fixed budgets—is specific to an SFT model trained on long-form teacher outputs from DeepSeek-R1. A critical stress-test would vary the SFT data: train one SFT checkpoint with the standard long-form DeepSeek-R1 responses (verbosity = high), another with responses truncated to the first 8K tokens (verbosity = medium), and another with responses generated by a teacher prompted to be concise (verbosity = low). Then apply the identical RLHF stage to all three and measure the reasoning improvement. If the paper's mechanism is correct, the gain should be largest for the high-verbosity SFT model and negligible for the low-verbosity SFT model. If instead RLHF provides reasoning gains independent of verbosity—for example, by teaching the model to structure its reasoning more logically, or by suppressing hallucinations that the reward model detects—the gain would be more uniform. This experiment would clarify whether RLHF's benefit is a general property of alignment training or a targeted fix for a specific SFT artifact, which determines whether RLHF-first should be a default recommendation or a conditional optimization.

Scale the execution-free SWE reward model to self-play or iterative improvement. The current SWE RL reward function requires ground-truth patches from human developers—it computes similarity between the model's generated patch and a known correct patch. This makes it suitable for RL on existing bug-fix datasets but not for scenarios where the model explores novel bugs or self-generates training data. A follow-up would train a dedicated reward model for SWE tasks that predicts patch correctness from code and issue descriptions alone, without requiring ground-truth patches. The training data exists: the paper already generates millions of patches with execution-based verification labels during SFT data construction (Section 3.3.2—DeepSeek-R1-0528 generates up to 8 responses per prompt, with correctness determined by Unidiff similarity and test execution). These labeled patches could train a SWE-specific reward model that replaces both the lexical similarity and semantic similarity components of the Cascade RL reward function. The key metric would be whether RL training with this learned reward model achieves comparable SWE-bench performance to the ground-truth-patch-based approach, which would unlock SWE RL on arbitrary repositories without human-written fixes. The paper's finding that larger reward models (72B) are more robust to distribution shift than smaller ones (Section 6.2, Figure 10) suggests that a 72B-scale SWE reward model would be the natural starting point.

Investigate whether Cascade RL's resistance to forgetting holds for domains with genuinely conflicting reward structures. The paper's theoretical argument (Section 4.1.1) acknowledges a vulnerability: "Catastrophic forgetting may still occur when the reward of a new domain sharply conflicts with that of a previous one (e.g., optimizing for concise responses versus detailed, step-by-step reasoning), particularly when prompts from different domains are semantically similar." The paper navigates this tension via the length-extension curriculum in Math RL—compressing reasoning first (24K stage), then extending it (32K, 40K stages)—but this is a mitigation, not a test of the boundary. A deliberate stress-test would introduce a domain whose reward directly punishes a behavior that a previous domain's reward encourages. For example: train a model with RLHF that strongly rewards conciseness (RM fine-tuned to prefer responses under 500 tokens), then apply a "Verbose Reasoning" RL stage that explicitly rewards response length on math problems. If the paper's argument about policy-dependent distribution is correct, the model should maintain conciseness on non-math prompts while extending math reasoning, since the prompts are semantically distinct and the policy continues to sample concise general-domain responses. If forgetting occurs, it would identify a boundary condition for Cascade RL and suggest that some domain pairs genuinely need joint training or explicit replay. AIME 2024/2025 and ArenaHard would serve as the primary metrics—tracking whether math reasoning improves without general alignment degrading.

Replicate Cascade RL on a non-Qwen base model family with different pretraining properties. The paper uses Qwen3-8B-Base and Qwen3-14B-Base exclusively. Qwen3 models have specific properties that may interact with Cascade RL: a 32K native context window (which constrains SWE RL prompt length, as shown in Table 13), a particular verbosity profile after SFT, and a specific base reasoning capability that determines which problems are solvable versus in the "hard problem ceiling." Replicating the full Cascade RL pipeline on a Llama-4, DeepSeek-V3, or Gemma base model would test whether the cascade ordering, the RLHF-first design, the length-extension curriculum, and the specific hyperparameters (learning rates, temperatures, rollout counts) transfer or need retuning. The paper's release of training data and recipes makes this replication feasible. The key metrics would be: (a) whether the same stage ordering produces the same pattern of results (RLHF improves reasoning, IF-RL temporarily degrades reasoning but recovers with later stages, Math RL transfers to coding, SWE RL does not degrade prior domains), and (b) whether hyperparameters optimized for Qwen3 transfer without modification. If they do, Cascade RL becomes a general-purpose post-training framework rather than a Qwen3-specific recipe. If they don't, the paper's findings are narrower than implied, and future work would need to characterize which model properties determine the optimal cascade configuration.

Practical Applications and Downstream Use Cases

Cost-efficient deployment of unified reasoning models in production APIs. The paper's unified 8B model (Nemotron-Cascade-8B) provides a concrete deployment advantage: it can handle both simple instruct queries and complex reasoning tasks from a single set of weights, eliminating the need to maintain separate thinking and non-thinking model endpoints. The per-turn /think and /no_think flags enable the routing logic to be determined by the application layer rather than the model infrastructure—a chat application can append /no_think to factual lookup queries (avoiding the token cost and latency of reasoning) and /think to multi-step problem-solving queries (activating deep reasoning when needed). The quantitative case: the unified 8B model matches the dedicated 8B-Thinking model on reasoning benchmarks (Table 8: AIME 2024 89.5 vs. 88.8, LCB v5 74.3 vs. 74.5) while substantially outperforming it on instruction-following (IFEval 90.2 vs. 83.7) and matching it on alignment (ArenaHard 87.9 vs. 85.8). This means a single deployment gets the best of both worlds—no capability tradeoff between thinking and non-thinking modes—while reducing serving infrastructure complexity by 50% compared to maintaining separate models. The paper's open release of model weights and training recipes makes this immediately actionable for organizations running their own inference infrastructure.

Accelerating RL training cycles by parallelizing domain-specific stages. The paper's finding that domain-wise RL stages are largely independent—later stages do not degrade earlier capabilities—has immediate implications for how RL training can be organized in multi-team settings. In a joint-training paradigm, all domain teams (math, code, SWE, alignment) must coordinate on a single training run, with cycle times determined by the slowest verification step (typically code execution or SWE Docker environments). In Cascade RL, once the RLHF foundation is established, Math RL, Code RL, and SWE RL can potentially be developed and optimized in parallel by different teams—Math RL on math prompts, Code RL on code prompts, each with its own hyperparameter tuning—and applied sequentially or combined via model merging. The paper shows that Math RL improves code benchmarks (Table 6: LCB v5 for 8B-Thinking goes from 69.0 to 71.2) and Code RL does not degrade math (Table 7: AIME 2024 drops only 1.9 points, within evaluation variance). This cross-domain transfer means teams do not need to wait for each other: a math team can iterate on Math RL recipes, a code team on Code RL recipes, and the results compose without destructive interference. For organizations training large models on multi-month cycles, this parallelization could substantially accelerate the iteration speed of post-training research.

Training specialized SWE agents from general-purpose reasoning checkpoints. The SWE RL results (Table 8: 14B-Thinking achieves 43.1% on SWE-bench Verified, 8B unified achieves 37.2%) demonstrate that a single stage of domain-specific RL added to a general-purpose reasoning model can produce SWE performance competitive with specialized models (DeepSWE-32B at 42.2%, SWE-agent-LM-32B at 40.2%). This has immediate practical relevance: organizations building coding agents can start from the open Nemotron-Cascade checkpoints rather than training from scratch, apply additional SWE RL on their proprietary repositories, and achieve competitive SWE performance with substantially less total training compute than training a dedicated SWE model from a base model. The paper's execution-free reward model design (Section 4.7.2) further reduces the infrastructure barrier—the reward can be computed without managing Docker containers, using an LLM judge (Kimi-Dev-72B) for semantic similarity scoring. A practical deployment would: (1) take the released Nemotron-Cascade-14B-Thinking checkpoint, (2) curate a SWE RL dataset from the organization's own repositories following the data construction pipeline in Section 3.3.2 (localization, repair, test generation prompts with DeepSeek-R1-0528 or a comparable teacher), (3) apply the SWE RL recipe from Section 4.7.2 (batch 128, rollout 16, learning rate 2.5e-6, 120 steps), and (4) deploy with the test-time scaling pipeline from Section 7.4 (best@k selection with regression and reproduction tests). The paper's SWE-bench numbers provide a baseline expectation: the 14B model gains ~3.5 points from SWE RL (39.6 → 43.1, Table 8 vs. Table 7), and test-time scaling to best@32 adds another ~10.7 points (43.1 → 53.8, Figure 12b).

Constructing multi-domain RL training curricula from the paper's per-stage recipes for new domains. The paper's Cascade RL framework—though demonstrated on five specific domains—provides a template for adding new domains without disrupting existing capabilities. The template is: (1) curate SFT data for the new domain (teacher-generated responses with mode-appropriate formatting), (2) design a domain-appropriate reward function (rule-based verification if possible, execution-based if tests exist, LLM-judge-based if neither is available), (3) construct RL prompts that are disjoint from both SFT data and other domains' RL data, (4) apply RL as a new final stage after all existing stages, using GRPO with domain-tuned hyperparameters, (5) evaluate on all previous benchmarks to verify no forgetting. The paper's results suggest that Step 5 will show minimal degradation (±1–2%) as long as the new domain's reward structure does not directly conflict with existing capabilities. A concrete example: adding a "formal proof" domain (e.g., Lean or Isabelle theorem proving) to the existing cascade. The SFT data would use DeepSeek-R1-0528 to generate proof sketches; the RL reward would use the formal proof assistant's kernel to verify correctness; the stage would be added after SWE RL. Based on the paper's finding that SWE RL—a similarly specialized final stage—caused no measurable degradation on math or code benchmarks (Table 8), a formal proof RL stage would be expected to improve theorem-proving capability without affecting AIME or LiveCodeBench performance. The paper's open-source training data and recipes make this extension practical: a researcher can take the released checkpoints, curate domain-specific data following the templates in Section 3.2, and run a single additional RL stage using the hyperparameter configurations in Appendix D.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Unified Reasoning Models Are Hard to Build

Why This Problem Matters

Where Prior Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Supervised Fine-Tuning: The Two-Stage Curriculum

The Unified Chat Template

General-Domain SFT Data Curation

Math Reasoning SFT Data

Code Reasoning SFT Data

Science Reasoning SFT Data

Tool Calling SFT Data

Software Engineering SFT Data

Post-SFT Results and the Baseline for Cascade RL

Reward Model Training for RLHF

The GRPO Training Algorithm

Stage 1: Reinforcement Learning from Human Feedback (RLHF)

Stage 2: Instruction-Following Reinforcement Learning (IF-RL)

Stage 3: Math RL

Stage 4: Code RL

Stage 5: SWE RL

The Localization Stage for SWE Evaluation

Test-Time Scaling for SWE

Why Cascade RL Resists Catastrophic Forgetting

4. Key Insights and Innovations

Innovation 1: Sequential Domain-Wise RL Is Resistant to Catastrophic Forgetting — and This Changes the Default Training Paradigm

Innovation 2: RLHF as a Reasoning Enhancer, Not Just an Alignment Tax

Innovation 3: The Unified Model Gap Can Be Closed by Training Non-Thinking Mode in Early Stages

Innovation 4: SWE RL Can Be Scaled Using Execution-Free Semantic Reward Models

Innovation 5: The Difficulty-Adaptive Length Extension Curriculum as a General RLVR Stabilizer

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Overall Pipeline Performance (Table 1)

Competitive Programming Results (Table 9, Figure 1)

IOI 2025 Results (Figure 5)

Per-Stage Progression (Tables 2, 4, 5, 6, 7, 8)

Difficulty-Dependent Code Improvements (Figures 7, 8)

SWE Test-Time Scaling (Figure 12)

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

No Head-to-Head Comparison Against Joint RL Training on the Same Base Model

The RLHF-Improves-Reasoning Effect Depends Critically on SFT Model Verbosity

Total Training Compute Cost Is Not Reported or Accounted For

Hardest Problems Remain Essentially Unsolved Across All Methods

Single Base Model Family and Limited Domain Coverage Constrain Generalizability Claims

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

Olmo 3

Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process