QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning

ArXiv: 2505.17667

🎯 Pitch

Supervised fine-tuning alone teaches long-context models to reason but paradoxically traps them in local optima that sabotage reinforcement learning. QWENLONG‑L1 breaks this deadlock by gradually scaling context length during RL and injecting the hardest examples back into training, pushing a 32B model past OpenAI‑o3‑mini to rival Claude‑3.7‑Sonnet‑Thinking across seven benchmarks.

1. Executive Summary

This paper introduces QWENLONG-L1, a reinforcement learning framework that adapts short-context large reasoning models to long-context scenarios through progressive context scaling—a training strategy combining curriculum-guided phased RL (gradually increasing input length from 20K to 60K tokens), difficulty-aware retrospective sampling (retaining the hardest examples from earlier phases to incentivize exploration), and a warm-up SFT initialization. Evaluated on seven long-context document QA benchmarks using R1-Distill-Qwen-14B/32B as base models, QWENLONG-L1-32B achieves an average 5.1-point gain over its base model, surpassing OpenAI-o3-mini and Qwen3-235B-A22B while matching Claude-3.7-Sonnet-Thinking—and QWENLONG-L1-14B surpasses DeepSeek-R1 at Pass@2 with only 2 samples (73.7 vs. 72.1). The work establishes that RL-driven long-context adaptation yields substantial reasoning gains through progressive context scaling, but that SFT alone—despite increasing reasoning behaviors like grounding and backtracking—fails to translate these behaviors into performance improvements, plateauing with only 0.8–3.2 point gains and leaving the model trapped in local optima that limit subsequent RL improvements when applied to long-context data.

2. Context and Motivation

The Core Problem: Extending Reasoning RL to Long-Context Scenarios

The paper addresses a fundamental gap in the current landscape of large reasoning models: reinforcement learning has been successfully applied to enhance reasoning capabilities in short-context tasks, but no comparable framework exists for long-context reasoning. This is not merely a matter of scaling up existing approaches—the paper's preliminary experiments (Figure 2) reveal that directly applying short-context reasoning RL to long-context inputs produces qualitatively different and significantly degraded training dynamics. The problem manifests in two specific failure modes:

• Suboptimal training efficiency: Reward scores converge more slowly in long-context settings compared to short-context counterparts. This is driven by a marked reduction in output entropy when models process long-context inputs, which restricts the exploratory behavior essential for effective policy optimization during RL.
• Unstable optimization: The training process exhibits intermittent spikes in KL divergence—sudden jumps where the policy model drifts sharply away from the reference model. These instabilities arise from the inherent variance amplification caused by longer output lengths combined with heterogeneous input length distributions, leading to greater variability during policy updates.

The paper's core insight is that these challenges are not incidental implementation difficulties but are fundamental to the nature of long-context reasoning itself. In short-context tasks, the model primarily relies on parametric knowledge stored in its weights during pretraining. Reasoning involves retrieving and manipulating this internal knowledge through extended chain-of-thought. In long-context reasoning, the model must perform an additional, non-trivial operation before reasoning: it must locate, extract, and ground relevant information from a potentially massive input context—which may include multiple documents, extensive financial reports, or sprawling legal filings—and then construct reasoning chains that faithfully incorporate this grounded information. This two-stage process (grounding + reasoning) introduces new sources of variance and complexity that short-context RL methods are not designed to handle.

Why This Gap Matters

The paper argues that bridging this gap is critical for both practical and theoretical reasons.

Practical impact: the deep research bottleneck. The paper explicitly situates long-context reasoning RL within the context of emerging "deep research" applications (Section 1). These are systems where an LLM must autonomously collect information from knowledge-intensive environments—reading and synthesizing multiple documents, financial reports, legal filings, scientific papers—and then reason across them to produce coherent analyses or answer complex questions. Examples cited include OpenAI's deep research feature [27], Gemini's deep research mode [38], and Perplexity's deep research capability [40]. All of these applications share a common requirement: the model must handle inputs spanning tens of thousands of tokens while maintaining robust contextual grounding and multi-step reasoning. Without principled methods for training models to perform this kind of long-context reasoning, these applications remain bottlenecked by the quality of their underlying reasoning engines.

The paper also argues that long-context reasoning is the natural target for deploying LRMs in real-world, information-intensive environments (Section 1, introduction paragraph). Tasks like analyzing a company's SEC filings to answer financial questions, synthesizing evidence across multiple legal documents, or tracing multi-hop relationships in large knowledge bases all require reasoning over substantially more context than the 4K-token regimes where current LRMs have demonstrated their strongest results [45, 22].

Theoretical significance: extending the RL-for-reasoning paradigm. From a research perspective, the gap matters because it tests whether the core idea underlying recent LRM breakthroughs—that RL can incentivize the emergence of sophisticated reasoning strategies (divide-and-conquer, backtracking, self-verification)—generalizes beyond the short-context settings where it has been demonstrated. The paper refers to these short-context successes as analogous to human "slow thinking" [4], where models learn to decompose problems, check their work, and revise their approaches. But these behaviors have primarily been observed and studied in tasks like mathematical problem-solving and competitive programming, where the reasoning operates on compact problem statements [24, 7, 47]. Whether RL can induce similar reasoning behaviors when the model must simultaneously manage long-range contextual dependencies is an open question with significant implications for the scope and limits of RL-driven reasoning improvement.

Prior Approaches and Their Limitations

To understand what QWENLONG-L1 contributes, it is necessary to trace the landscape the paper positions itself within and identify where existing approaches fall short.

Short-context reasoning RL has advanced rapidly—but remains siloed. The paper acknowledges a rich body of recent work on using RL to enhance reasoning in language models (Section 1, first paragraph). This includes: OpenAI-o1 [26, 15], which demonstrated that RL-trained models can produce extended chains of thought showing deliberate reasoning; DeepSeek-R1 [11], which showed that RL alone (without supervised fine-tuning on reasoning traces) can elicit reasoning behaviors; Qwen-QwQ [41, 43], which explored RL for reflection and self-correction; and numerous algorithmic innovations including GRPO [34], DAPO [54], and process-level reward modeling [3, 55]. These works collectively established that RL is a powerful tool for improving reasoning quality on tasks like mathematics (DeepScaler [24], Light-R1 [45]), coding (Open R1 [8], competitive programming [7]), and logical reasoning (Logic-RL [47]). However, the paper notes that all of these advances have been demonstrated in short-context settings—typically with inputs under 4K tokens. The extension to long-context scenarios "remains unexplored" (Section 1, paragraph 2).

Group-relative RL algorithms exist but have not been validated for long-context training. The paper explicitly adopts group-relative policy optimization methods—GRPO [34] and DAPO [54]—rather than standard PPO [33] with a learned value function. The motivation (Section 2.2) is practical: the quadratic complexity of attention mechanisms in long-context inputs makes training a separate value network computationally prohibitive. GRPO and DAPO circumvent this by estimating advantages through group-normalized rewards (Equation 5: $A_{i,t} = \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)}$), eliminating the need for a value network entirely. However, these algorithms were developed and validated in short-context settings. The paper's contribution is not the algorithms themselves but their application within a progressive context scaling framework that makes them viable for long-context training—and the empirical demonstration that they work in this regime.

Long-context pretraining methods exist for language modeling—but not for reasoning RL. The paper draws inspiration from prior work on extending context length during pretraining, citing Fu et al. [9], Xiong et al. [48], and Gao et al. [10] (Section 1, paragraph introducing QWENLONG-L1). These works showed that progressively increasing context length during language model pretraining—rather than training directly on long contexts—enables more stable and efficient context extension. The paper adapts this core idea to the RL setting, but the adaptation is non-trivial: pretraining context extension operates on next-token prediction with well-behaved cross-entropy loss surfaces, while RL context extension must contend with non-stationary reward signals, exploration-exploitation tradeoffs, and the risk of policy collapse. The paper's progressive context scaling strategy (Section 2.3) is the first to apply curriculum-based context lengthening to reinforcement learning for reasoning.

Supervised fine-tuning for long-context reasoning has been attempted but shows limited gains. While not extensively surveyed in the paper's related work section, the experimental results in Table 4 provide direct evidence about the efficacy of SFT-based approaches. The paper applies SFT to its base models using high-quality reasoning traces distilled from DeepSeek-R1, constructing 5.3K short-context (20K token limit) question-document-answer triplets. Despite this careful data curation, the SFT models achieve only modest improvements: +0.8 points on R1-Distill-Qwen-14B and +3.2 points on R1-Distill-Qwen-32B (Table 4, SFT rows). Furthermore, these gains are inconsistent across benchmarks—the 14B SFT model actually loses performance on DocMath (-1.0) and Qasper (-2.5). This aligns with findings from the data selection literature [59, 19] suggesting that SFT's imitation learning paradigm tends to prioritize superficial pattern matching over substantive reasoning skill development, particularly when the training data distribution does not perfectly match the deployment distribution.

More revealing is the experiment in Section 4.3 (Figure 6), where the paper tests a long-context SFT model trained on 10K context examples. This model surpasses the short-context SFT baseline by 2.1 points, showing that longer-context SFT does provide some benefit. However, when RL is subsequently applied to this long-context SFT model, it achieves only an additional 0.3 points of improvement—compared to 3.2 points when RL is applied to the short-context SFT model. The paper interprets this as evidence that excessive SFT "traps models in local optima" that constrain RL's ability to explore and improve, and establishes a core principle: SFT provides acceptable performance with less effort, but RL is indispensable for optimal performance (Section 4.3, first analysis).

Process reward models and search-based methods for long-context tasks are unaddressed. The paper's hybrid reward mechanism (Section 2.4) uses a combination of rule-based verification and LLM-as-a-judge to produce binary outcome rewards. It does not employ process reward models (PRMs) that score intermediate reasoning steps, which have been central to search-based approaches on short-context math tasks (Lightman et al., 2023). The paper does not discuss whether PRMs could be trained for long-context reasoning or whether search algorithms like beam search over a PRM would help. This is an acknowledged limitation gap rather than a failure of prior work—but it means the paper operates in a regime (outcome-only rewards, no search) that may not represent the full space of possible long-context reasoning methods.

How This Paper Positions Itself

QWENLONG-L1 positions itself as the first RL framework specifically designed for long-context reasoning (Section 1, contributions list, bullet 2). This is not an incremental extension of existing methods—the paper argues it fills a recognized gap that no prior work has addressed. The positioning is built on several strategic choices that differentiate it from both short-context reasoning RL and long-context pretraining approaches.

From short-context proficiency to long-context generalization. The paper frames its core task as a transition problem: given an LRM that has already been trained (via distillation from a reasoning model like DeepSeek-R1) to perform strong short-context reasoning, how do you extend this capability to long-context inputs? This framing is important because it acknowledges that the base models (R1-Distill-Qwen-14B and -32B) already possess non-trivial reasoning abilities—they just need to be adapted. The paper does not claim to teach reasoning from scratch on long contexts; rather, it provides a training recipe for transferring existing reasoning skills to the long-context regime.

A training framework, not a single algorithm. QWENLONG-L1 is not presented as a new RL algorithm but as a framework—an integrated set of training strategies (warm-up SFT, curriculum-guided phased RL, difficulty-aware retrospective sampling) that work together to enable stable long-context RL optimization. This is explicit in the paper's title and throughout the text: "QWENLONG-L1, a framework that adapts short-context LRMs to long-context scenarios via progressive context scaling" (Abstract). The framework is algorithm-agnostic in principle; the paper demonstrates it with both GRPO and DAPO, and both variants show substantial gains (Table 4: +4.1 and +4.0 on 14B, +5.1 and +4.7 on 32B).

Document QA as a testbed for long-context reasoning. The paper chooses document question answering (DocQA) as its evaluation domain, and this choice is strategic (Section 3 introduction). DocQA inherently requires both the grounding capability (locating relevant information in long documents) and the reasoning capability (synthesizing that information to answer complex questions) that the paper argues are the defining features of long-context reasoning. Within DocQA, the paper constructs DOCQA-RL-1.6K, a training dataset spanning three reasoning types—mathematical reasoning (DocMath), logical reasoning (synthesized multiple-choice questions), and multi-hop reasoning (MultiHopRAG, Musique)—to ensure the framework learns generalizable long-context reasoning rather than overfitting to a single task format.

Progressive context scaling as the key enabler. The paper positions progressive context scaling—not the choice of RL algorithm, not the specific reward function, not the base model architecture—as its primary enabling innovation. The ablation studies in Section 4.2 (Figure 5) are designed to isolate and validate the contributions of each component of this strategy. The paper shows that: (1) curriculum-guided phased RL substantially outperforms naive single-stage RL across all configurations (GRPO, DAPO, SFT+GRPO, SFT+DAPO); (2) the mechanism of improvement is the stabilization of training dynamics—phased RL exhibits lower KL divergence fluctuations and prevents the entropy collapse seen in single-stage training (Figure 5c); and (3) difficulty-aware retrospective sampling further enhances performance by retaining hard examples that maintain high policy entropy and incentivize continued exploration (Figure 5d).

Difficult questions as a target, not a special case. A distinguishing feature of the paper's approach is its treatment of hard examples. In many RL-for-reasoning works, hard examples are simply the ones where the model has the most room to improve—they are implicitly the target. QWENLONG-L1 operationalizes this through its difficulty-aware retrospective sampling strategy (Section 2.3, Equation 10). Difficulty is explicitly quantified as the inverse mean reward ($\text{diff}(x, c) = 1 / \text{mean}(\{r_i\}_{i=1}^G)$), and the hardest examples (those with an average accuracy of zero) are carried forward from earlier training phases into later ones. This ensures the model continues to encounter and explore challenging instances rather than progressively training on easier examples that would reduce entropy and stall improvement.

Relative to existing work on training dynamics. The paper's preliminary analysis (Figure 2) comparing short-context and long-context RL training dynamics is positioned as a diagnostic contribution: it identifies why naive approaches fail, not just that they fail. The four sub-figures (reward convergence delay, entropy reduction, KL divergence spikes, output length variance) provide a mechanistic explanation that motivates each component of the proposed framework. This is a deliberate positioning choice—the paper wants to be seen as not just proposing a solution, but also diagnosing the problem in a way that guides future work.

Practical and actionable. The paper positions QWENLONG-L1 as a practical training recipe, not a theoretical proposal. All training details are specified (Section 3.2): 32×A100-80G GPUs, batch size 128, mini-batch size 32, rollout number 8, learning rate 2e-6, temperature 0.7, top-p 0.95, max output length 10K. The two-phase curriculum (20K → 60K input length) is concrete and reproducible. The datasets (DOCQA-RL-1.6K, SFT dataset) are described with statistics in Table 3. The evaluation benchmarks and configurations are fully specified. This positions the work as immediately reproducible and deployable, consistent with the paper's stated goal of providing "a fundamental recipe and practice for long-context reasoning optimization" (Section 1, contributions list, bullet 3).

The Conceptual Gap: What Makes Long-Context Reasoning RL Different

To fully appreciate the paper's contribution, it is worth articulating exactly what changes when the input includes a long context, and why existing short-context methods fail. The paper provides the formalization in Equations 1 and 2 (Section 2.1), but the implications are broader than the formalism alone.

In short-context reasoning RL (Equation 1), the policy model $\pi_\theta$ conditions on a question $x$ and generates an answer $y$ using parametric knowledge. The key property is that the information needed to answer the question is either already in the model's weights or can be derived through pure logical operations on knowledge stored in the weights. The variability in the output $y$ comes primarily from the model's reasoning strategy—whether it tries different approaches, backtracks when stuck, verifies intermediate steps, etc.

In long-context reasoning RL (Equation 2), the policy model additionally conditions on a context $c$. Crucially, the correct answer depends on information in $c$, not (only) on parametric knowledge. The model must first locate the relevant information within $c$—a task that is itself non-trivial when $c$ spans tens of thousands of tokens and may contain irrelevant, distracting, or contradictory information—and then construct a reasoning chain that correctly uses this grounded information. This introduces several new sources of complexity:

1. Input-dependent difficulty: The difficulty of a question now depends not just on the reasoning required but also on the structure of the context—how long it is, how the relevant information is distributed, whether there are distractors, whether information from multiple locations must be synthesized. This means the same question asked with different contexts can have different effective difficulty, and the model must adapt accordingly.

2. Grounding errors as a new failure mode: In short-context reasoning, failures are typically reasoning errors (logical mistakes, calculation errors, failure to backtrack). In long-context reasoning, a new failure mode appears: grounding errors, where the model extracts incorrect or irrelevant information from the context, or correctly identifies relevant information but misinterprets it, or fails to find critical information altogether.

3. Interaction between grounding and reasoning: These are not independent stages. Grounding errors propagate into reasoning (the model reasons correctly about wrong information), and reasoning strategies affect grounding (a model that backtracks might realize it grounded the wrong information and correct itself). This interaction creates feedback loops that can amplify variance in RL training—a small initial grounding error can lead to a very different trajectory than correct grounding, producing high-variance rewards.

4. Length heterogeneity: Long-context inputs vary dramatically in length—from ~5K tokens (Qasper) to ~176K tokens (DocMath max length) in the paper's evaluation benchmarks (Table 3). The output length also varies substantially depending on the complexity of reasoning required. This heterogeneity means the policy sees training examples with vastly different computational requirements, making it difficult to find a single learning rate or clipping threshold that works well across all examples.

5. Entropy dynamics: The paper's key empirical finding (Figure 2b) is that long-context inputs cause a more rapid reduction in output entropy during RL training. This means the policy becomes less exploratory more quickly, potentially converging to suboptimal strategies before it has had a chance to discover better ones. The paper hypothesizes that this is because long contexts provide more surface-level cues that the model latches onto (specific document structures, recurring patterns), reducing the diversity of its outputs even when those surface-level cues do not reliably lead to correct answers.

The paper's framework addresses each of these challenges: curriculum-guided phased RL creates a controlled progression of input lengths to manage heterogeneity; difficulty-aware retrospective sampling ensures hard examples from earlier phases continue to challenge the model; warm-up SFT provides a robust initialization that prevents the model from developing bad grounding habits early in RL training; and the hybrid reward function handles the answer format diversity that arises from grounding-based reasoning (same semantic answer expressed differently depending on which specific text was grounded).

3. Technical Approach

This is primarily a systems and training methodology paper whose core idea is that stable RL-driven adaptation from short-context to long-context reasoning can be achieved by progressively scaling the input length during training, combined with mechanisms to maintain exploration difficulty and provide a robust initialization.

3.1 Reader Orientation

QWENLONG-L1 is a training framework—a recipe of sequential stages and adaptive data strategies—that takes an existing large reasoning model proficient at short-context tasks and retrains it to perform complex, multi-step reasoning over documents spanning tens of thousands of tokens. The framework solves the problem that naive reinforcement learning on long-context inputs produces unstable optimization (KL divergence spikes) and inefficient training (entropy collapse, slow reward convergence) by structuring the RL process as a curriculum that gradually increases context length, while strategically retaining hard examples and providing a supervised warm-up phase to establish foundational grounding capabilities before RL begins.

3.2 Big-Picture Architecture (Diagram in Words)

The QWENLONG-L1 system has five major components arranged in a sequential pipeline:

1. Base Model (R1-Distill-Qwen-14B/32B): A pretrained, short-context reasoning model that serves as the starting point. This model already possesses strong reasoning capabilities (chain-of-thought, backtracking, verification) but has been trained exclusively on short-context tasks like mathematics and coding.

2. Warm-Up SFT Stage: The base model undergoes supervised fine-tuning on high-quality reasoning traces distilled from DeepSeek-R1, using short-context (≤20K tokens) document-question-answer triplets. This establishes a robust initial policy that can ground information from documents and produce properly formatted reasoning chains before exposure to the instability of RL. The output is the SFT model, which serves as the initial policy $\pi_\theta$ for RL.

3. Curriculum-Guided Phased RL: The RL training is divided into discrete phases with progressively increasing input length targets. Phase I operates on inputs ≤20K tokens; Phase II operates on inputs ≤60K tokens. Within each phase, the policy model generates groups of responses, receives hybrid rule-based + LLM-judge rewards, and is updated using group-relative advantage estimation (via GRPO or DAPO).

4. Difficulty-Aware Retrospective Sampler: As training transitions from Phase I to Phase II, a subset of the hardest examples from Phase I (those with near-zero average reward) are retained and mixed into Phase II training data. Difficulty is quantified as the inverse of the mean group reward. This prevents the training distribution from becoming progressively easier and maintains exploration pressure.

5. Hybrid Reward Function: Each generated response receives a binary reward computed as the maximum of a rule-based exact-match check against the gold answer and an LLM-as-judge semantic equivalence evaluation. The rule-based component ensures precision for answers with canonical forms; the LLM judge handles answer diversity (paraphrases, different units, equivalent expressions).

Information flows as follows: base model → warm-up SFT on short-context data → Phase I RL on ≤20K inputs with difficulty scoring → difficulty-aware sampler retains hardest examples → Phase II RL on ≤60K inputs augmented with retained hard examples → final long-context LRM.

3.3 Roadmap for the Deep Dive

• First, the RL objective formalization: Understanding exactly how the paper extends the standard KL-regularized RL objective to include a long context $c$—this establishes the mathematical problem being solved and reveals why long-context RL differs fundamentally from short-context RL.
• Second, the RL algorithms (GRPO and DAPO): How the paper adapts group-relative policy optimization methods to avoid training a value network (which would be computationally prohibitive for long contexts), including the specific advantage estimation formulas, clipping strategies, and the modifications DAPO introduces to prevent entropy collapse and length explosion.
• Third, the progressive context scaling strategy: The core training architecture—curriculum-guided phased RL, difficulty-aware retrospective sampling, and warm-up SFT—explaining how each component addresses the specific failure modes identified in Figure 2.
• Fourth, the hybrid reward mechanism: How the paper combines rule-based verification and LLM-as-a-judge to produce reliable binary rewards for long-context DocQA tasks where answer formats are diverse and exact string matching alone would produce excessive false negatives.
• Fifth, the training dataset and configuration: What data is used, how it is constructed, and the specific hyperparameters that make the framework reproducible.

3.4 Detailed, Sentence-Based Technical Breakdown

Formalizing the Long-Context Reasoning RL Objective

The paper begins by extending the standard RL objective for language models to explicitly incorporate a long context. In the standard formulation, the policy model $\pi_\theta$ takes a question $x$ as input and generates an answer $y$. The optimization maximizes expected reward while penalizing deviation from a reference policy $\pi_{\text{ref}}$:

$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot | x)} [r_\phi(x, y)] - \beta \mathbb{D}_{\text{KL}} [\pi_\theta(y | x) || \pi_{\text{ref}}(y | x)]$

where $\mathcal{D}$ is the training distribution of questions, $r_\phi(x, y)$ is the reward for output $y$ given input $x$, and $\beta$ controls the strength of the KL divergence penalty that keeps the policy from drifting too far from the reference model.

What this equation represents: The objective balances two forces—maximizing the expected reward (encouraging the policy to produce correct, high-quality answers) and minimizing the KL divergence from a reference model (preventing the policy from collapsing to degenerate outputs that exploit reward signal without actually learning reasoning). The expectation over $x \sim \mathcal{D}$ means we want good average performance across the training distribution; the expectation over $y \sim \pi_\theta$ means we are optimizing over the policy's own generated outputs (on-policy learning).

Why this form: The KL penalty is essential in RL for language models because without it, the policy can rapidly diverge from the pretrained distribution, producing nonsensical outputs that happen to score well under whatever reward function is used (a phenomenon known as reward hacking). The $\beta$ parameter trades off between aggressive optimization and staying close to the pretrained model.

The long-context extension: The paper's key formal move is replacing the input $x$ with a pair $(x, c)$, where $c$ is a supporting context document that may span tens of thousands of tokens:

$\max_{\pi_\theta} \mathbb{E}_{x, c \sim \mathcal{D}, y \sim \pi_\theta(\cdot | x, c)} [r_\phi(x, c, y)] - \beta \mathbb{D}_{\text{KL}} [\pi_\theta(y | x, c) || \pi_{\text{ref}}(y | x, c)]$

where $x$ is the question text, $c$ is the supporting context (a long document or set of documents), and $y$ is the model's reasoning chain and answer. The policy now conditions on both question and context, and must ground relevant information from $c$ before reasoning.

Operationally, what changes: The reference model $\pi_{\text{ref}}$ is now evaluated with the context included in its input—so deviations are penalized relative to what the base model would produce given the same context, not what it would produce without context. The reward function $r_\phi(x, c, y)$ can now evaluate correctness conditioned on the specific context, which is critical because the same question may have different correct answers depending on what document $c$ contains.

Why this formalization matters: This equation makes explicit that the policy must learn to process $c$—not merely to extract a surface-level pattern but to ground its reasoning in the specific information the context provides. The KL penalty now operates on the joint distribution over $(x, c)$, meaning the policy cannot simply memorize context-independent answer templates—it must learn to condition its outputs on the actual content of $c$.

The prompt template (Table 1) concatenates context and question as:

Please read the following text and answer the question below.
<text> context </text> question
Format your response as follows: "Therefore, the answer is (insert answer here)".

This template structures the input so the model can distinguish between the document to be read and the question to be answered, and forces the output into a format that enables reliable answer extraction for reward computation.

Group-Relative RL Algorithms: GRPO and DAPO

The paper's core technical challenge in selecting an RL algorithm is that training a value network—required by standard PPO—is computationally prohibitive for long-context inputs. Value networks must process the same long input sequences as the policy, and the quadratic complexity of self-attention makes this infeasible at the sequence lengths involved (up to 60K tokens during training, with evaluation extending to 120K tokens). The paper therefore adopts group-relative policy optimization, where advantages are estimated by comparing rewards within a group of different outputs generated for the same input, rather than through a learned value function.

GRPO: Group Relative Policy Optimization

GRPO [34] eliminates the value network entirely by estimating each output's advantage through group-level normalization. For a given input $(x, c)$, the old policy model $\pi_{\theta_{\text{old}}}$ generates a group of $G$ complete outputs $\{y_i\}_{i=1}^G$. Each output receives a scalar reward $r_i$ from the reward function described in Section 2.4. The optimization objective for the current policy $\pi_\theta$ is:

$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{x, c \sim \mathcal{D}, \{y_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot | x, c)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min \left( \frac{\pi_\theta(y_{i,t} | x, c, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} | x, c, y_{i,<t})} A_{i,t}, \text{clip} \left( \frac{\pi_\theta(y_{i,t} | x, c, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} | x, c, y_{i,<t})}, 1 - \varepsilon, 1 + \varepsilon \right) A_{i,t} \right) - \beta \mathbb{D}_{\text{KL}}(\pi_\theta || \pi_{\text{ref}}) \right]$

where $G$ is the group size (number of outputs generated per input), $|y_i|$ is the length of the $i$-th output in tokens, $y_{i,t}$ is the $t$-th token of the $i$-th output, $\pi_\theta(\cdot | x, c, y_{i,<t})$ is the policy's predicted probability for token $y_{i,t}$ given the input and preceding tokens, $\pi_{\theta_{\text{old}}}$ is the frozen policy used for sampling, $\varepsilon$ is the clipping hyperparameter, and $A_{i,t}$ is the token-level advantage.

The advantage estimator: The key innovation of GRPO is replacing the learned advantage function (typically requiring a value network) with a simple group-level z-score:

$A_{i,t} = \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)}$

where $\text{mean}(\{r_i\}_{i=1}^G)$ is the average reward across all $G$ outputs for this input, and $\text{std}(\{r_i\}_{i=1}^G)$ is the standard deviation of those $G$ rewards.

What this computes: The advantage for every token in output $i$ is identical and equals how many standard deviations above or below the group mean that output's reward falls. If an output is the best in its group (reward above the mean), all its tokens receive a positive advantage—the policy is encouraged to make those token choices more likely. If an output is below average, all its tokens receive a negative advantage—the policy is discouraged from those choices. The division by standard deviation normalizes the advantage magnitude across inputs with different reward scales.

Why identical per-token advantages work: This is a critical design choice—GRPO assigns the same advantage $A_{i,t}$ to every token in output $i$, regardless of which tokens were responsible for the output's success or failure. This may seem crude, but it works in practice because: (1) the policy gradient for a sequence is proportional to the sum of token-level advantages, and scaling all tokens by the same factor preserves the direction of the gradient while weighting sequences by their outcome quality; (2) token-level credit assignment in long reasoning chains is extremely noisy, and attempting to assign per-token advantages without a value network would likely be more harmful than helpful; (3) the clipping mechanism prevents any single token from dominating the update.

The clipping mechanism: The $\min(\rho_t A_{i,t}, \text{clip}(\rho_t, 1 - \varepsilon, 1 + \varepsilon) A_{i,t})$ term, where $\rho_t = \frac{\pi_\theta(y_{t} | x, c, y_{<t})}{\pi_{\theta_{\text{old}}}(y_{t} | x, c, y_{<t})}$ is the probability ratio, serves the same purpose as in standard PPO: it prevents the policy from changing too much in a single update by capping how much the probability of any single token can increase or decrease. If a token's probability under the new policy is more than $1 + \varepsilon$ times its probability under the old policy, the gradient is clipped to prevent further increase. If it is less than $1 - \varepsilon$ times the old probability, the gradient is clipped to prevent further decrease.

The paper removes the KL term: In their implementation, the authors "remove the KL term in the GRPO objective to encourage the exploration capacity of the policy model, following the common suggestions in recent works" (Section 2.2, GRPO paragraph). This means the $-\beta \mathbb{D}_{\text{KL}}(\pi_\theta || \pi_{\text{ref}})$ term is omitted. This is justified by the observation that the clipping mechanism and the group-relative advantage estimation together provide sufficient regularization to prevent policy collapse, and that removing the KL penalty gives the policy more freedom to explore novel reasoning strategies not present in the reference model's output distribution.

Operationally: For each input, the old policy generates $G = 8$ candidate outputs (the rollout number of 8 specified in Section 3.2). Each output receives a binary reward (0 or 1) from the hybrid reward function. The rewards are z-scored within the group, producing an advantage for each output. The policy is then updated to increase the probability of tokens in high-advantage outputs and decrease the probability of tokens in low-advantage outputs, subject to the clipping constraint.

DAPO: Decoupled Clip and Dynamic Sampling Policy Optimization

DAPO [54] extends GRPO with four modifications designed to address specific pathologies observed in group-relative RL training. The objective becomes:

$\mathcal{J}_{\text{DAPO}}(\theta) = \mathbb{E}_{(x,c) \sim \mathcal{D}, \{y_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot | x, c)} \left[ \frac{1}{\sum_{i=1}^G |y_i|} \sum_{i=1}^G \sum_{t=1}^{|y_i|} \min \left( \frac{\pi_\theta(y_t | x, c, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, c, y_{<t})} A_{i,t}, \text{clip} \left( \frac{\pi_\theta(y_t | x, c, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, c, y_{<t})}, 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}} \right) A_{i,t} \right) \right]$

Modification 1: Asymmetric clipping with higher upper bound. Unlike GRPO's symmetric clip range $[1 - \varepsilon, 1 + \varepsilon]$, DAPO uses separate lower and upper clip thresholds $\varepsilon_{\text{low}}$ and $\varepsilon_{\text{high}}$, with $\varepsilon_{\text{high}} > \varepsilon_{\text{low}}$. This "higher clip threshold" is specifically designed to "avoid entropy collapse" (Section 2.2, DAPO paragraph). In GRPO, when the clip range is too tight, the policy cannot increase the probability of good tokens fast enough relative to the rate at which other tokens are penalized, causing the output distribution to concentrate on a small set of tokens and collapse entropy. The higher upper clip allows the policy to more aggressively increase probabilities of tokens in high-advantage sequences, maintaining diversity.

Modification 2: Dynamic sampling—filtering examples with zero reward variance. Before computing advantages, DAPO checks whether the $G$ outputs for a given input all have the same reward (either all 0 or all 1). If so, the advantage for every output is zero (the numerator $r_i - \text{mean}(\{r_i\})$ is zero), meaning this input contributes nothing to the gradient. DAPO "removes examples with zero reward variance" (Section 2.2) from the training batch entirely, saving computation and preventing the model from wasting capacity on inputs where the policy's outputs are either all correct or all incorrect—cases where no relative improvement signal exists.

Modification 3: Token-level loss normalization. GRPO averages the loss per-output (dividing by $G$, then by $|y_i|$), which gives equal weight to each output regardless of its length. DAPO instead divides by the total number of tokens across all outputs $\sum_{i=1}^G |y_i|$, giving equal weight to each token. This is designed to "mitigate the length bias" (Section 2.2)—preventing the policy from being overly influenced by very long outputs that dominate the gradient under output-level averaging. Without token-level normalization, the model might learn to produce excessively long outputs simply because they contribute more to the loss.

Modification 4: Overlong reward shaping with a soft length penalty. DAPO introduces a length-aware penalty that modifies the reward before computing advantages:

$r_{\text{final}_i} = \begin{cases} r_i, & |y_i| \leq L_{\text{max}} - L_{\text{cache}} \\ r_i + \frac{(L_{\text{max}} - L_{\text{cache}}) - |y_i|}{L_{\text{cache}}}, & L_{\text{max}} - L_{\text{cache}} < |y_i| \leq L_{\text{max}} \\ r_i - 1, & L_{\text{max}} < |y_i| \end{cases}$

where $L_{\text{max}}$ is the maximum allowed sequence length, $L_{\text{cache}}$ is a buffer zone for gradual penalization, and $|y_i|$ is the length of output $i$.

What this computes: Outputs within the safe zone ($|y_i| \leq L_{\text{max}} - L_{\text{cache}}$) receive their original reward unchanged. Outputs in the buffer zone receive a linearly decreasing penalty—the further into the buffer, the smaller the added term (which is negative since $(L_{\text{max}} - L_{\text{cache}}) - |y_i| \leq 0$ in this region). Outputs exceeding $L_{\text{max}}$ receive a harsh $-1$ penalty, which when combined with a binary reward (0 or 1) can push the final reward as low as $-1$.

Why this shape: The penalty structure provides a soft constraint rather than a hard truncation. Hard truncation at $L_{\text{max}}$ would create a discontinuity in the reward surface, which is problematic for gradient-based optimization—outputs just below and just above the cutoff would receive dramatically different rewards despite being nearly identical. The buffer zone creates a smooth gradient that pushes the policy toward shorter outputs without creating a cliff. The harsh penalty for exceeding $L_{\text{max}}$ ensures that outputs beyond the maximum length are strongly discouraged while still being visible in the training signal (unlike hard truncation, where they would simply not be scored).

The advantage estimator in DAPO uses the modified rewards:

$A_{i,t} = \frac{r_{\text{final}_i} - \text{mean}(\{r_{\text{final}_i}\}_{i=1}^G)}{\text{std}(\{r_{\text{final}_i}\}_{i=1}^G)}$

This is identical in form to GRPO's advantage but operates on the length-penalized rewards.

Why two algorithms? The paper evaluates both GRPO and DAPO to demonstrate that the progressive context scaling framework provides benefits independent of the specific RL algorithm. Both algorithms show substantial improvements (Table 4: +4.0 and +4.1 on 14B for GRPO and DAPO respectively; +4.7 and +5.1 on 32B), with DAPO showing a slight edge on the 32B model. The choice between them may depend on specific training characteristics—DAPO's additional mechanisms make it more robust to length explosion and entropy collapse, but GRPO's simplicity may be advantageous in resource-constrained settings.

Progressive Context Scaling: The Core Training Architecture

The paper's central methodological contribution is a three-component strategy for enabling stable RL training on long-context inputs. This section details each component—warm-up SFT, curriculum-guided phased RL, and difficulty-aware retrospective sampling—explaining both their implementation and the evidence for why they are necessary.

Warm-Up Supervised Fine-Tuning

Before any RL training begins, the base model undergoes supervised fine-tuning on high-quality demonstration data. The motivation is threefold: (1) the base models (R1-Distill-Qwen-14B/32B) are trained on short-context reasoning tasks and may not reliably produce properly formatted outputs when given long document contexts; (2) RL training is inherently unstable, and a policy that starts with poor grounding capabilities will generate mostly incorrect outputs with zero reward, providing no learning signal; (3) SFT provides a known-good initialization that ensures the policy has basic competence before RL begins to optimize it.

Training data construction: The SFT dataset $\mathcal{D}_{\text{SFT}}$ consists of 5,305 high-quality question-document-answer triplets distilled from DeepSeek-R1 [11] (Table 3). Each example contains a question $x$, a supporting context $c$, and a gold-standard reasoning path $y^*$ including both the chain-of-thought and the final answer. The data is filtered for "quality, complexity, and diversity" (Section 2.3, SFT paragraph), and the documents are controlled for both length and precision of contextual information. Critically, the input length is capped at $L_1 = 20$K tokens—the same length as Phase I of the subsequent RL curriculum—to align SFT with the first RL phase.

Training objective: Standard next-token prediction with teacher forcing:

$\mathcal{L}_{\text{SFT}}(\theta) = -\mathbb{E}_{(x,c,y^*) \sim \mathcal{D}_{\text{SFT}}} \frac{1}{|y^*|} \sum_{t=1}^{|y^*|} \log \pi_\theta(y_t^* | x, c, y_{<t}^*)$

where $\pi_\theta(y_t^* | x, c, y_{<t}^*)$ is the model's predicted probability for the $t$-th token of the gold reasoning path given the input and preceding gold tokens.

What it computes: The average negative log-likelihood of the correct next token at each position in the reasoning trace, averaged over all tokens and all training examples. Minimizing this objective means making the gold reasoning path as likely as possible under the model's output distribution.

Why full-sequence teacher forcing: Rather than only training on the final answer or using a selective loss, SFT trains on every token of the reasoning trace—including the chain-of-thought that leads to the answer. This forces the model to learn not just what the correct answer is, but also how to reason toward it: which information to extract from the document, how to structure multi-step reasoning, and how to format the final answer. The length normalization ($\frac{1}{|y^*|}$) ensures that longer reasoning traces do not dominate the loss simply because they have more tokens.

Training configuration: SFT is conducted on 32×A100-80G GPUs for 3 epochs, with a batch size of 128, a learning rate of 5e-6, and an input length cap of 20K tokens (Section 3.2, SFT Training paragraph).

Evidence for necessity: The ablation in Figure 5(a) shows that across all RL configurations (GRPO, DAPO, phased RL, with/without retrospective sampling), adding warm-up SFT produces significant performance improvements. Figure 5(b) reveals the mechanism: SFT not only provides a higher starting reward but also sustains lower gradient norm during RL training, indicating more stable updates. The paper interprets this as SFT handling the "format alignment" problem—teaching the model the basic template of long-context reasoning (read document, extract information, reason step-by-step, output answer in required format)—so that RL can focus on optimizing reasoning quality rather than learning basic task structure from sparse rewards.

Why not more or longer-context SFT? The paper explicitly tests this in Section 4.3 (Figure 6). A long-context SFT model trained on 10K examples with context up to the full training length achieves better standalone performance than the short-context SFT model (+2.1 points). However, when RL is applied to this long-context SFT model, it gains only 0.3 additional points, compared to 3.2 points when RL is applied to the short-context SFT model. The paper concludes that extensive long-context SFT "traps models in local optima" —the policy converges to a good but not optimal strategy that RL cannot easily escape because the gradients become too small. A short-context SFT initialization leaves more "room" for RL to discover better strategies.

Curriculum-Guided Phased Reinforcement Learning

The core of QWENLONG-L1's training architecture is dividing the RL process into $K$ discrete phases, each constrained to a specific range of input lengths. The paper uses $K = 2$ phases:

• Phase I: Input length ≤ 20K tokens
• Phase II: Input length ≤ 60K tokens

The transition condition is based purely on input length: during phase $k$, the policy is trained exclusively on examples where:

$L_{k-1} < |x| + |c| \leq L_k, \quad L_0 = 0$

where $|x|$ is the question length in tokens and $|c|$ is the context length in tokens, with $L_1 = 20,000$ and $L_2 = 60,000$.

What this means operationally: In Phase I, the training data consists only of examples where the combined question and context length is between 0 and 20K tokens. The policy learns to handle reasoning over short-to-medium documents. In Phase II, the policy is trained only on longer examples (20K–60K tokens). By the end of Phase II, the policy has been exposed to the full range of input lengths up to 60K tokens.

Why a two-phase curriculum: The paper's preliminary experiment (Figure 2) demonstrates that directly training on the full range of input lengths produces two pathologies: entropy collapse (the policy's output distribution becomes overly narrow, reducing exploration) and KL divergence instability (the policy drifts erratically from the reference model). The phased curriculum addresses both by structuring the learning process:

1. Entropy preservation: By starting with shorter contexts, the policy learns fundamental grounding and reasoning skills in a regime where the variance in input structure is lower and the information density is higher (shorter documents have less irrelevant material to distract the model). This allows the policy to develop robust strategies before facing the additional complexity of longer documents. When Phase II begins, the policy already has a strong basis of reasoning behaviors, so the entropy reduction observed in Figure 2(b) is mitigated—the policy has more "stored entropy" from Phase I that carries into Phase II.

2. KL stabilization: The KL divergence spikes in Figure 2(c) are attributed to the high variance in input length—the policy sees some very short and some very long inputs in the same batch, requiring very different computational strategies and producing very different gradient magnitudes. By homogenizing input lengths within each phase, the phased curriculum reduces the variance of policy updates, smoothing the KL divergence trajectory.

Evidence from ablation: Figure 5(a) shows that phased RL outperforms single-stage RL across all configurations (GRPO, DAPO, SFT+GRPO, SFT+DAPO). Figure 5(c) provides the mechanistic explanation: single-stage RL exhibits "heightened instability, as demonstrated by fluctuating KL divergence and entropy collapse," while phased RL maintains more stable KL divergence and higher entropy throughout training. The improvement from phased RL is less pronounced when the policy is initialized with SFT—suggesting that warm-up SFT "partially compensates for curriculum design" (Section 4.2)—but is still positive.

The transition between phases: The paper does not describe a gradual blending between phases; Phase I uses exclusively ≤20K examples, Phase II uses exclusively >20K examples. However, the difficulty-aware retrospective sampling (described next) ensures that some of the hardest examples from Phase I are carried into Phase II, creating a form of continuity.

Difficulty-Aware Retrospective Sampling

When transitioning from Phase I to Phase II, a naive approach would simply discard all Phase I training examples and train only on the longer Phase II data. The paper argues this is suboptimal because Phase I contains hard examples that the policy has not yet mastered—removing them from the training distribution removes the pressure to continue improving on them, potentially allowing the policy to regress on difficult short-context examples.

Difficulty quantification: The difficulty of an example $(x, c)$ is defined as the inverse of the mean reward across a group of outputs:

$\text{diff}(x, c) = \frac{1}{\text{mean}(\{r_i\}_{i=1}^G)}$

where $\{r_i\}_{i=1}^G$ are the rewards for $G$ outputs generated by the base model (before RL training) for input $(x, c)$.

What this computes: If the base model's average reward on an example is low (near 0), the difficulty is high (approaching infinity). If the average reward is high (near 1), the difficulty is low (approaching 1). Examples where the model always gets zero reward have infinite difficulty, making them the highest-priority for retention.

The sampling mechanism: The paper states that it implements "importance sampling weighted by difficulty scores to curate retrospective instances" and specifically "maintain[s] the most difficult samples with an average accuracy of zero from phase I to II" (Section 3.2, RL Training paragraph). This means examples from Phase I where the base model achieved exactly zero reward across all 8 rollouts are guaranteed to be included in Phase II training, ensuring the policy continues to encounter and attempt to solve problems it initially found impossible.

Why this matters for exploration: Figure 5(d) shows that these retained hard examples exhibit "significantly lower reward and higher policy entropy" during Phase II training. High entropy means the policy is uncertain about what to do—it is still exploring different strategies rather than having converged to a single (possibly incorrect) approach. This uncertainty provides a gradient signal for improvement: when the policy eventually discovers a correct approach for one of these hard examples, the reward increase is large, and the advantage signal is strong. Without retrospective sampling, Phase II would consist entirely of examples that are hard because of their length, not because of inherent reasoning difficulty—the policy might learn to handle length but lose the pressure to solve genuinely hard reasoning problems.

Operational details: The paper uses difficulty scores computed from the base model (before any RL training), not from the current policy during training. This means the difficulty ranking is fixed and does not adapt as the policy improves. The rationale is computational efficiency—recomputing difficulty from the current policy at each phase transition would require generating groups of outputs from the intermediate policy, which is expensive. However, this fixed ranking means some examples that were initially hard may become easy during Phase I training, and they would still be carried forward. The paper does not discuss whether this causes any negative effects.

Hybrid Reward Mechanism

The reward function is the sole training signal during RL—it determines which outputs are considered good and which are not. For long-context DocQA tasks, designing a reward function is challenging because: (1) answers can be expressed in multiple semantically equivalent forms (e.g., "$980,000" vs. "980 thousand dollars" vs. "$0.98 million"); (2) strict exact-match penalties would produce false negatives that incorrectly penalize correct reasoning; (3) but purely model-based evaluation is more expensive and may introduce its own biases.

The paper's solution is a hybrid reward that takes the maximum of two evaluations:

$r_\phi(x, y) = \max(r_{\text{rule}}(y), r_{\text{LLM}}(x, y))$

Rule-Based Verification ($r_{\text{rule}}$): The generated output $y$ is scanned for a final answer using regular expressions aligned with the prompt template's required format ("Therefore, the answer is (insert answer here)"). The extracted answer $y_{\text{ans}}$ is compared against the gold answer $y_{\text{gold}}$ using exact string matching:

$r_{\text{rule}}(y) = \mathbb{I}(y_{\text{ans}} = y_{\text{gold}})$

where $\mathbb{I}$ is the indicator function—1 if the strings match exactly, 0 otherwise.

What this captures: Exact, literal correctness. If the model produces the answer in exactly the canonical form, it gets the reward. This is highest precision—there are essentially no false positives (a string match is almost certainly correct).

What this misses: Any semantically equivalent but syntactically different answer: different number formats ("32.4" vs. "32.40"), different unit expressions ("$980,000" vs. "980000 dollars"), different wordings ("July 26, 2023" vs. "26 July 2023"), or answers that require interpretation (the model says "approximately 32 million" when the gold answer is "32,400,000").

Intentional omission of format reward: The paper explicitly notes that they "omit the format reward for answer extraction, as the base model demonstrates sufficient inherent format compliance capabilities" (Section 2.4, Rule-Based Verification paragraph). Many RL-for-reasoning works include a small bonus reward for producing outputs that match the expected format (e.g., containing a properly formatted final answer box). The paper argues that adding a format reward would "oversimplify the learning objective, potentially hindering the model's ability for reasoning chain exploration"—the model might learn to produce properly formatted but incorrect answers, exploiting the format reward while avoiding the harder task of actually solving the problem.

LLM-as-a-Judge ($r_{\text{LLM}}$): A separate language model evaluates semantic equivalence:

$r_{\text{LLM}}(x, y) = \text{LLM}(x, y_{\text{ans}}, y_{\text{gold}})$

where the LLM judge receives the original question $x$, the model's extracted answer $y_{\text{ans}}$, and the gold answer $y_{\text{gold}}$. It produces a binary correctness score (YES/NO) based on the prompt template in Table 2, which instructs it to "determine if two answers are equivalent, without attempting to solve the original problem" and to "compare the answers to verify they represent identical values or meaning, even when written in different forms or notations."

The judge model: The paper uses Qwen2.5-1.5B-Instruct with temperature 0 for deterministic scoring. The choice of a small model is deliberate—it "enables efficient reward computation during online RL training while maintaining evaluation reliability" (Section 2.4, Combined Reward paragraph). A 1.5B-parameter judge is cheap enough to run on every generated output during RL training without becoming the computational bottleneck.

Why maximum rather than average or AND: The $\max$ operator means an output is considered correct if either the rule-based match or the LLM judge says it is correct. This design choice reflects the relative risks: false negatives (correct answer marked wrong) are more harmful for RL training than false positives (incorrect answer marked right), because false negatives suppress good exploration by penalizing the policy for producing correct-but-differently-formatted answers. The $\max$ ensures that if the LLM judge recognizes an answer as semantically equivalent to the gold answer, it receives credit even if the string match fails. The rule-based component serves as a backstop: for answers with canonical forms, the exact match provides a high-confidence signal that doesn't depend on a potentially fallible LLM judge.

The judge's prompt (Table 2):

You are an expert in verifying if two answers are the same. Your input is a problem and two answers, Answer 1 and Answer 2. You need to check if they are equivalent. Your task is to determine if two answers are equivalent, without attempting to solve the original problem. Compare the answers to verify they represent identical values or meaning, even when written in different forms or notations. Your output must follow the following format:

1) Provide an explanation for why the answers are equivalent or not.
2) Then provide your final answer in the form of: [[YES]] or [[NO]]

Problem: question
Answer 1: predicted answer
Answer 2: gold answer

The judge is explicitly instructed not to solve the problem—it should only compare the two answers as stated. This prevents the judge from "filling in" for model errors by independently solving the problem and recognizing that the model's wrong answer would lead to a different result.

Operational flow: During RL training, each generated output $y$ undergoes answer extraction to produce $y_{\text{ans}}$. The extracted answer is checked against the gold answer via exact string match. If it matches, $r_{\text{rule}} = 1$ and the reward is 1 without needing the LLM judge (computational savings). If it does not match, the LLM judge is invoked to evaluate semantic equivalence. If the judge returns YES, the reward is 1; if NO, the reward is 0. The final reward is 1 if either path succeeds, and 0 only if both fail.

Potential weaknesses inherent in this design: (1) The LLM judge, despite being instructed not to solve the problem, may still be biased toward marking plausible-sounding answers as correct—this is the familiar LLM-as-judge reliability issue. The paper mitigates this with temperature 0 (deterministic) and a small model that is less capable of "filling in" missing reasoning. (2) The $\max$ operator means that if either evaluator has a systematic bias toward marking answers as correct, the combined reward will inherit that bias. The paper does not report the agreement rate between the rule-based and LLM judge components. (3) For evaluation, the paper uses DeepSeek-V3 rather than Qwen2.5-1.5B-Instruct as the judge model, indicating that the training-time judge is a compromise between accuracy and computational cost. The paper does not analyze whether training with a weaker judge and evaluating with a stronger one introduces any systematic gap.

Training Dataset and Configuration

This section details the concrete data and hyperparameters that operationalize the QWENLONG-L1 framework. All configurations are drawn from Section 3.

DOCQA-RL-1.6K: The RL training dataset. The paper constructs a specialized dataset of 1,591 document question-answering problems spanning three reasoning categories:

• Mathematical Reasoning (600 problems): Drawn from DocMath [57], a dataset requiring numerical reasoning over long, specialized documents such as financial reports. The paper samples 75% of items from each DocMath subset for training and reserves 25% for evaluation (Section 3.1, footnote 2). These problems require the model to extract numerical values from dense financial text and perform calculations—as illustrated in the case studies where the model must compute total capital costs from bond offering details.

• Logical Reasoning (600 problems): Synthesized multiple-choice questions generated by DeepSeek-R1 [11]. These require logical analysis of real-world documents spanning "legal, financial, insurance, and production domains from our curated collection" (Section 3.1). The synthesis process involves having DeepSeek-R1 generate questions that test logical deduction from provided documents—the questions are not pre-existing benchmarks but purpose-built for training long-context logical reasoning.

• Multi-Hop Reasoning (400 problems): 200 examples from MultiHopRAG [36] and 200 from Musique [44]. These require synthesizing information across multiple documents or multiple sections of a single document to answer a question that cannot be answered from a single location. Multi-hop reasoning is particularly challenging because the model must maintain and connect information retrieved from different parts of the context.

Dataset statistics (Table 3): The RL dataset has an average length of 11,437 tokens and a maximum of 59,559 tokens. This means most training examples are well within the Phase I limit (≤20K), with a tail of longer examples requiring Phase II. The SFT dataset is larger (5,305 examples) but shorter on average (13,064 tokens) with a maximum of only 20,003 tokens—consistent with its role as a short-context warm-up.

RL training configuration (Section 3.2):

• Hardware: 32×A100-80G GPUs
• Batch size: 128 (train batch size, meaning 128 distinct inputs per update)
• Mini-batch size: 32 (gradient accumulation across 4 mini-batches before each update)
• Rollout number ($G$): 8 (8 candidate outputs generated per input)
• Learning rate: 2e-6
• Sampling temperature: 0.7 (controls output diversity—higher than typical greedy decoding to encourage exploration)
• Top-p: 0.95 (nucleus sampling parameter—only tokens comprising the top 95% of probability mass are considered)
• Maximum output length: 10K tokens
• Phases: Two phases, $L_1 = 20$K tokens, $L_2 = 60$K tokens

Why these sampling parameters: The temperature of 0.7 and top-p of 0.95 represent a moderate exploration setting. Temperature > 1.0 would produce more uniform token distributions (maximum exploration but lower quality); temperature < 0.5 would produce sharper distributions (higher average quality but limited diversity). The value 0.7 balances exploration (needed for RL to discover good strategies) with exploitation (needed for the base model's existing capabilities to surface). The top-p of 0.95 is a standard choice that truncates the long tail of very unlikely tokens without overly restricting the model.

The interaction between rollout number and group-relative advantage: With $G = 8$ rollouts per input, the advantage estimator (Equation 5) computes z-scores across 8 rewards. For binary rewards (0 or 1), this means the advantage can take at most 8 distinct values. For example, if 0 of 8 outputs are correct, all advantages are zero (no learning signal). If 1 of 8 is correct, that output gets a high positive advantage and the 7 incorrect outputs get a small negative advantage. If all 8 are correct, all advantages are zero (no learning signal—the policy already handles this input perfectly). The useful cases are intermediate: when some outputs are correct and others are not, the contrast provides a training signal. This is why the dynamic sampling in DAPO filters out examples with zero variance—they contribute no gradient regardless of batch composition.

The maximum output length constraint: 10K tokens is substantial—it allows the model to produce multi-thousand-word reasoning chains, as illustrated in the case studies where outputs span 2,000–5,000 tokens. The DAPO soft length penalty (Equation 7) applies to outputs approaching this limit, discouraging unnecessarily verbose reasoning while still allowing complex multi-step reasoning when needed.

SFT training configuration (Section 3.2):

• Hardware: 32×A100-80G GPUs
• Epochs: 3
• Batch size: 128
• Learning rate: 5e-6
• Input length: 20K tokens

The higher learning rate for SFT: The SFT learning rate (5e-6) is higher than the RL learning rate (2e-6). This is consistent with standard practice: supervised fine-tuning on high-quality demonstrations is a more stable optimization problem (fixed targets, well-behaved loss surface) and can tolerate higher learning rates. RL training is more delicate (non-stationary rewards, on-policy sampling, risk of policy collapse) and requires more conservative updates.

Evaluation configuration (Section 3.3):

• Maximum input length: 120K tokens (substantially longer than the 60K training maximum—testing generalization to unseen context lengths)
• Maximum output length: 10K tokens
• Sampling temperature: 0.7
• Top-p: 0.95
• Judge model: DeepSeek-V3 with temperature 0.0
• Scoring: Maximum of exact match and LLM-judged accuracy, matching the training reward function
• Test-time scaling: For Pass@K evaluation, 16 candidate outputs are generated per input question

The 120K evaluation context length: This is a deliberate stress test—the model is trained on contexts up to 60K tokens but evaluated on contexts up to 120K tokens (Table 3: DocMath max length 176,285; Frames max length 117,131). This tests whether the progressive context scaling strategy produces genuine length generalization (the ability to process longer contexts than seen during training) or merely memorizes the training length distribution. The paper does not explicitly analyze length generalization, but the strong evaluation results on benchmarks with maximum lengths exceeding the training maximum suggest some degree of generalization.

Benchmark details (Table 3): The seven evaluation benchmarks span:

• DocMath [57]: Document-based mathematical reasoning, average length 17,645 tokens, max 176,285
• Frames [18]: Fact retrieval and reasoning, average 15,756, max 117,131
• 2WikiMultihopQA [13]: Multi-hop QA over Wikipedia, average 7,530, max 17,035
• HotpotQA [51]: Multi-hop reasoning, average 13,431, max 17,640
• Musique [44]: Multi-hop question composition, average 16,327, max 17,883
• NarrativeQA [17]: Reading comprehension over book/movie narratives, average 29,887, max 65,357
• Qasper [5]: QA over NLP papers, average 5,074, max 21,927

The benchmarks vary substantially in average length (from ~5K to ~30K tokens) and maximum length (from ~17K to ~176K tokens), testing both the model's ability to handle very long inputs and its robustness to the specific document types and reasoning formats in each benchmark.

Why these benchmarks collectively test long-context reasoning: Each benchmark requires the model to: (1) locate information within long documents (grounding), (2) synthesize information from potentially multiple locations (multi-hop for 2Wiki, HQA, Musique), (3) reason about the extracted information (mathematical calculation for DocMath, logical deduction for the logical reasoning benchmarks), and (4) produce a concise answer despite having processed a long context. This combination of grounding, synthesis, reasoning, and compression is what the paper defines as long-context reasoning.

Design Choices and Their Justifications: A Summary

The paper makes several non-obvious design choices that are worth highlighting explicitly:

• Two phases, not continuous scaling: The paper uses two discrete phases (20K → 60K) rather than a continuous increase in context length. This simplifies implementation and allows clear phase boundaries for retrospective sampling. A continuous curriculum might provide smoother training but would require continuous difficulty tracking and more complex data management.

• 20K → 60K, not 20K → 120K: The training maximum (60K) is half the evaluation maximum (120K). This means the model must generalize to context lengths it never saw during RL training. The paper's results suggest this generalization occurs, but the mechanism is not explored—possible explanations include the model's pretraining on long documents (R1-Distill-Qwen models may have seen long contexts during pretraining) or the progressive scaling teaching generalizable grounding strategies rather than length-specific ones.

• SFT only on short contexts: The paper deliberately restricts SFT to the Phase I length range (≤20K) even though it could have created longer SFT examples. The Figure 6 analysis shows this was a good choice—longer-context SFT provides better standalone performance but severely limits RL improvements. This is a counterintuitive finding that challenges the natural assumption that more SFT on the target distribution is always better.

• Binary rewards, no partial credit: The reward function produces only 0 or 1—there is no partial credit for partially correct reasoning, correct reasoning with incorrect final answer, or correct answer with flawed reasoning. This sparse reward signal means the model must discover correct answers through exploration to receive any positive signal. The group-relative advantage structure partially mitigates this by providing comparative signal even when no outputs are correct (if some outputs are "closer" to correct in ways the LLM judge can recognize), but the fundamental sparsity remains a challenge addressed by the progressive scaling strategy (shorter contexts are easier, providing more positive reward signal early in training).

• Qwen2.5-1.5B as judge for training, DeepSeek-V3 for evaluation: This asymmetry means the training reward function is a noisier approximation of the true correctness signal. Any systematic differences between how the small judge and the large judge evaluate answers create a distribution shift between training and evaluation. The paper's strong results suggest this shift is manageable, but it is an implicit assumption that small-model judgment quality is sufficient for training-time reward computation.

• Removing the KL penalty: The paper explicitly omits the KL divergence term from the GRPO objective to "encourage exploration capacity." This is a departure from the standard RLHF formulation where KL regularization is considered essential. The paper justifies it by reference to "common suggestions in recent works" and implicitly relies on the clipping mechanism and group-relative advantages to provide sufficient regularization. The ablation studies show this works empirically for this setup, but it increases the risk of policy collapse in different settings (different base models, different reward structures, different data distributions).

4. Key Insights and Innovations

Innovation 1: Long-Context Reasoning RL as a Distinct Paradigm Requiring Its Own Training Methodology

The paper's most fundamental conceptual contribution is not a specific algorithm but the formal recognition that long-context reasoning reinforcement learning constitutes a fundamentally different optimization problem from short-context reasoning RL—a distinction the field had not previously articulated or addressed. Prior to this work, the dominant assumption (implicit in essentially all RL-for-reasoning research) was that methods successful on short-context tasks—mathematics, coding, logic puzzles with compact problem statements—would transfer naturally to longer inputs by simply concatenating the context to the prompt. The standard RL objective (Equation 1) treats all inputs as interchangeable tokens fed to the same policy optimization procedure.

The paper dismantles this assumption through a diagnostic contribution that is arguably more valuable than any single architectural component: the training dynamics comparison in Figure 2. By conducting a controlled experiment comparing RL training on short-context versus long-context inputs with otherwise identical setup, the paper surfaces two specific, mechanistically distinct failure modes that are invisible in short-context training:

• Entropy collapse: Long-context inputs systematically drive the policy's output distribution toward lower entropy faster than short-context inputs. This is more than a quantitative difference—it represents a qualitative shift in the exploration-exploitation tradeoff, where the model latches onto surface-level patterns in long documents (recurring formats, standard phrasing) that provide spurious certainty, prematurely shutting down the exploratory behavior that RL depends on to discover good reasoning strategies.

• KL divergence instability: The heterogeneous input length distribution of long-context data (ranging from ~5K to ~176K tokens even within a single benchmark, Table 3) amplifies variance in policy updates, producing intermittent spikes where the policy drifts sharply from the reference model. This is not merely noisier training—it represents a regime where gradient estimates become unreliable, and standard clipping mechanisms (designed for relatively homogeneous input distributions) fail to provide adequate regularization.

What makes this contribution intellectually distinctive is that it defines the problem space before proposing solutions. The field had extensive empirical evidence that RL works for reasoning (OpenAI-o1, DeepSeek-R1, QwQ) and extensive evidence that language models can process long contexts (LongLoRA, RingAttention, various position encoding extensions), but no one had systematically examined what happens at the intersection. The paper's preliminary experiment (Figure 2) fills this gap by demonstrating that the intersection is not merely additive—long-context reasoning RL exhibits dynamics qualitatively different from either long-context pretraining or short-context reasoning RL alone. This reframes the research question from "can we apply RL to long-context tasks?" (which might yield a yes/no answer with ad-hoc engineering) to "what specific mechanisms must a training framework include to counteract the entropy collapse and KL instability pathologies that emerge uniquely in the long-context RL regime?"

The significance of this diagnostic framing extends beyond the paper's own solution. It provides a framework for evaluating future long-context RL methods: any proposed approach can be assessed by whether it addresses the two identified failure modes, and training dynamics curves (reward convergence, entropy trajectory, KL divergence) become diagnostic tools rather than just monitoring metrics. Prior work on RL training stability (e.g., the extensive PPO debugging literature) had focused on general issues like reward scaling and advantage normalization without identifying context-length-dependent pathologies. The paper's contribution here is to introduce input length as a first-class variable in RL stability analysis—a dimension that had been overlooked because short-context settings made it invisible.

Innovation 2: The Counterintuitive SFT-RL Tradeoff—More SFT Can Be Worse

The paper's second major conceptual contribution is the empirical finding that extensive supervised fine-tuning on the target distribution actively harms subsequent reinforcement learning, a result that inverts the natural intuition that better initialization always helps. This is established through the experiment in Figure 6 (Section 4.3), which compares three initialization strategies: (1) RL applied directly to the base model, (2) RL applied to a short-context SFT model (20K tokens), and (3) RL applied to a long-context SFT model (trained on 10K examples at full context length).

The results are striking and counterintuitive: the long-context SFT model achieves the best standalone performance (surpassing the short-context SFT model by 2.1 points and the base model by 2.6 points), yet when RL is applied, it gains only 0.3 additional points—compared to 3.2 points when RL is applied to the short-context SFT model. The model that received more supervised training on exactly the target distribution ends up dramatically worse after RL optimization than the model that received less.

This phenomenon—which the paper characterizes as SFT "trapping models in local optima"—represents a fundamental insight about the relationship between imitation learning and reinforcement learning for reasoning tasks. The standard pipeline in LLM post-training is: SFT on high-quality demonstrations, then RL to optimize further. This paper demonstrates that this pipeline has a sweet spot: too little SFT and the policy lacks basic competence, producing no positive reward signal; too much SFT and the policy's output distribution becomes too narrow, constraining the exploration that RL needs to discover better strategies.

What makes this intellectually significant beyond the specific numbers is that it challenges a widely held assumption in the RL-for-language-models community. The dominant narrative—exemplified by works like LIMA [59] and the emphasis on high-quality SFT data in DeepSeek-R1's training recipe [11]—has been that better SFT data and more extensive SFT training are uniformly beneficial. The paper's finding introduces a crucial qualification: SFT and RL serve complementary but potentially antagonistic purposes. SFT provides format alignment and basic competence quickly and cheaply, but it does so by narrowing the policy's output distribution toward a specific set of demonstrated behaviors. RL needs a broad distribution to explore beyond those demonstrations. The key implication is that SFT should be treated as a calibrated intervention—enough to establish a foundation, not so much that it precludes RL-driven discovery.

The paper's behavior analysis (Figure 7) deepens this insight by revealing the mechanism. Both SFT and RL increase the frequency of desirable reasoning behaviors (grounding, subgoal setting, backtracking, verification). However, the paper observes that "while SFT models demonstrate increased reasoning behaviors, these adjustments fail to transform into performance improvements" (Section 4.3). This suggests that SFT learns superficial patterns—the model produces more text that looks like reasoning, but the reasoning is not actually effective. RL, by contrast, refines the output distribution so that these behaviors, when they occur, are more likely to lead to correct answers. This is a specific instantiation of a broader principle: imitation learning can reproduce the form of good outputs without capturing their function, while outcome-based RL directly optimizes the function.

This finding has immediate practical implications that extend beyond long-context reasoning. Any project using RL to improve reasoning models should consider: is their SFT phase long enough to establish basic competence but short enough to leave room for RL? The paper provides a concrete diagnostic: if applying RL to a model yields only small gains, the model may be over-SFT'd. The metric isn't SFT performance—it's the delta between SFT and SFT+RL—and maximizing the delta may require deliberately limiting SFT.

Innovation 3: Progressive Context Scaling as a General Strategy for Distributionally-Challenging RL

The concept of curriculum learning—training on easier examples before harder ones—is well-established in machine learning. The concept of progressively increasing context length during pretraining has been explored in works like Fu et al. [9] and Xiong et al. [48]. The paper's innovation is not the idea of curriculum or the idea of context extension, but the specific synthesis of these ideas in the RL setting, combined with the diagnostic recognition that input length heterogeneity is the root cause of training instability, not merely a data management convenience.

Prior context extension work in pretraining operates in a supervised learning regime: the loss is well-behaved cross-entropy, the targets are fixed, and there is no exploration-exploitation tradeoff. The curriculum primarily serves to reduce computational cost (shorter sequences are cheaper) and to avoid catastrophic forgetting of short-context capabilities. In RL, the purpose of a context-length curriculum is fundamentally different: it is a stabilization mechanism that reduces the variance of policy gradients by homogenizing the computational requirements within each training batch. This isn't about cost savings or forgetting—it's about making the optimization landscape navigable.

What makes this distinctive is the mechanism of action. The paper demonstrates (Figure 5c) that single-stage RL on the full input length distribution exhibits entropy collapse and KL divergence fluctuation, while phased RL maintains higher entropy and more stable KL divergence. The mechanism is not simply that shorter contexts are easier to learn from (the traditional curriculum learning argument). Rather, it is that mixing very different input lengths in the same batch produces gradient updates with very different magnitudes and directions, and the resulting variance overwhelms standard RL stabilization mechanisms (clipping, advantage normalization). By grouping inputs of similar length, phased RL ensures that each update is based on a coherent batch where all examples require similar amounts of computation, making the average gradient more representative and reducing destructive interference.

This reframing has implications beyond context length. Any RL training scenario where input difficulty varies dramatically—tasks with heterogeneous complexity, multi-task RL with very different task horizons, RL with variable-length action sequences—could potentially benefit from a similar phased approach organized around the relevant difficulty dimension. The paper's contribution is to identify input length heterogeneity as a specific, measurable, and practically important source of RL instability and to demonstrate that a simple phased curriculum directly addresses it.

The difficulty-aware retrospective sampling component adds a crucial nuance that prevents the curriculum from becoming too easy. A naive phased curriculum would mean that later phases train only on longer (and potentially harder because of length) examples, but the paper's difficulty metric (inverse mean reward, Equation 10) captures a different dimension: reasoning challenge independent of input length. By carrying forward the hardest examples from earlier phases—specifically those where the model achieves zero average reward—retrospective sampling ensures that the training distribution in later phases includes both long-context challenges and genuinely difficult reasoning problems, maintaining the exploration pressure needed for continued improvement. Figure 5d confirms that these retained examples maintain lower reward and higher entropy, providing continued learning signal. This is a non-obvious design choice: the natural tendency would be to discard "easy" (short) examples in later phases, but the paper recognizes that "easy to read" is not the same as "easy to solve."

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on seven long-context DocQA benchmarks: DocMath [57] (numerical reasoning over financial documents, 200 test examples, max length ~176K tokens), Frames [18] (fact retrieval and reasoning, 824 examples, max ~117K tokens), 2WikiMultihopQA [13] (multi-hop QA over Wikipedia, 200 examples, max ~17K tokens), HotpotQA [51] (multi-hop reasoning, 200 examples, max ~18K tokens), Musique [44] (multi-hop question composition, 200 examples, max ~18K tokens), NarrativeQA [17] (reading comprehension over book/movie narratives, 200 examples, max ~65K tokens), and Qasper [5] (QA over NLP papers, 200 examples, max ~22K tokens). For 2WikiMultihopQA, HotpotQA, Musique, NarrativeQA, and Qasper, the specific data splits from LongBench [2] are used (Section 3.3, Benchmarks paragraph). The DocMath test split uses 25% of items from each subset, with the remaining 75% used for training (Section 3.1, footnote 2).

• Base model(s). All experiments use R1-Distill-Qwen-14B and R1-Distill-Qwen-32B [11] as the starting models. These are chosen because they represent strong short-context reasoning models that already exhibit chain-of-thought, backtracking, and verification behaviors from their distillation training, but have been trained exclusively on short-context mathematics, coding, and scientific reasoning tasks. The paper explicitly excludes the 7B and 1.5B distillation variants "due to their mathematical reasoning feature inherent from Qwen2.5-Math" (Section 3.2, footnote 3)—these smaller models were distilled from a mathematics-specialized teacher and would not provide a fair baseline for general long-context reasoning.

• Metrics. The primary metric is accuracy, reported as the fraction of test questions for which the selected final answer matches the ground truth. Accuracy is computed as the maximum of exact match and LLM-judged correctness: max(exact_match, LLM_judge_score). Exact match uses string comparison after extracting the final answer via regular expressions (aligned with the training reward function in Equation 12). LLM judging uses DeepSeek-V3 [21] with temperature 0.0 to assess semantic equivalence between the predicted answer and the gold answer (Section 3.3, Benchmarks paragraph). This dual-evaluation protocol mirrors the hybrid reward function used during training (Section 2.4) and is intended to avoid penalizing semantically correct answers expressed in non-canonical forms. For test-time scaling experiments, Pass@K is computed by generating 16 candidate outputs per question and reporting the fraction of questions where at least one of the top-K outputs (by the same max(exact_match, LLM_judge) criterion) is correct (Section 4.1, Additional Enhancements paragraph).

• Baselines. The paper compares against six proprietary and six open-source LRMs. Proprietary: OpenAI-o1-preview [15], Claude-3.7-Sonnet-Thinking [1], OpenAI-o3-mini [28], Qwen3-Plus [42], QwQ-Plus [43], and Gemini-2.0-Flash-Thinking [37]. Open-source: DeepSeek-R1 [11], Qwen3-235B-A22B [42], Qwen3-32B [42], QwQ-32B [43], R1-Distill-Qwen-32B [11], and R1-Distill-Qwen-14B [11]. Additionally, the paper reports two intermediate baselines from its own training pipeline: R1-Distill-Qwen-14B-SFT and R1-Distill-Qwen-32B-SFT (the base models after warm-up SFT but before RL), to isolate the contribution of RL beyond supervised fine-tuning. All baselines except the SFT models are evaluated zero-shot with temperature 0.7 and top-p 0.95 (Section 3.3, Configurations paragraph). Proprietary models with limited context length are evaluated with a maximum input length of 50K tokens, while QWENLONG-L1 models and open-source baselines are evaluated with a maximum input length of 120K tokens.

• Generation budget / compute accounting. The paper does not use a standardized FLOPs-based budget for comparing methods. Instead, all models are evaluated under a fixed generation protocol: each model generates a single output per question (Pass@1) with a maximum output length of 10K tokens. For test-time scaling (Pass@K), 16 candidate outputs are generated per question, with the Pass@K metric measuring the best-of-K performance. The paper explicitly notes that no additional search, ensembling, or majority voting is applied to the baseline models—each baseline produces a single answer per question under the same zero-shot protocol. This means the comparison evaluates the quality of a single reasoning chain from each model, not the models' ability to leverage test-time compute through sampling or search. For proprietary models with context length limits below 50K tokens, inputs are truncated, which may disadvantage those models on benchmarks with very long documents (DocMath max length 176K, Frames max length 117K).

• Cross-validation / statistical protocol. No cross-validation, bootstrapping, or statistical significance testing is reported. All results in Table 4 are single-point estimates computed over the full test sets (200–824 questions per benchmark). The paper does not report confidence intervals, standard deviations, or p-values for any comparison. For the ablation studies (Figure 5), results are reported as single accuracy values without error bars. The absence of statistical testing means that differences smaller than ~1 point on individual benchmarks (with test sets of ~200 questions) should be interpreted with caution—they could reflect sampling noise rather than genuine performance differences.

Main Quantitative Results

Overall Performance Across Seven Benchmarks (Table 4)

The headline result is that QWENLONG-L1-32B-DAPO achieves an average accuracy of 70.7% across the seven benchmarks, matching Claude-3.7-Sonnet-Thinking (70.7%) and surpassing OpenAI-o3-mini (70.4%), Qwen3-235B-A22B (70.6%), and DeepSeek-R1 (72.1%—but see the Pass@K discussion below for why this appears to be contradicting the text's claim). The improvement over the base R1-Distill-Qwen-32B model (65.6%) is 5.1 percentage points with DAPO and 4.7 points with GRPO, representing a substantial relative improvement from a model that already possessed strong reasoning capabilities.

Examining specific benchmark-level results reveals a nuanced pattern. On DocMath (mathematical reasoning over long documents): QWENLONG-L1-32B-GRPO achieves 68.0%, outperforming all proprietary baselines including Claude-3.7-Sonnet-Thinking (67.5%) and OpenAI-o3-mini (66.5%), and matching the best open-source baseline (Qwen3-235B-A22B at 67.5%). On 2WikiMultihopQA (multi-hop reasoning): QWENLONG-L1-32B-DAPO achieves 90.5%, matching QwQ-32B (90.5%) and trailing only Qwen3-235B-A22B (91.5%). On NarrativeQA (reading comprehension over long narratives): QWENLONG-L1-32B-GRPO achieves 61.0%, which lags behind OpenAI-o1-preview (68.0%), Claude-3.7-Sonnet-Thinking (61.5%), and QwQ-Plus (62.0%). On Qasper (QA over NLP papers): QWENLONG-L1-32B-DAPO achieves 58.5%, which is the highest among all models (ahead of OpenAI-o1-preview at 57.0% and QwQ-32B at 57.5%).

The performance profile is not uniform across benchmarks. QWENLONG-L1-32B shows its strongest relative performance on Qasper (58.5%, +8.5 points over base) and Musique (69.0%, +8.0 points over base), and its weakest relative performance on 2WikiMultihopQA (90.5%, +6.5 points, but high absolute performance) and HotpotQA (83.0%, +2.5 points). This heterogeneity suggests that the RL training is more effective for certain reasoning types—particularly those requiring information synthesis from dense technical documents (Qasper, DocMath)—than for others.

Comparing the two RL algorithms: DAPO slightly outperforms GRPO on the 32B model (70.7 vs. 70.3 average), but the difference is small (0.4 points) and varies by benchmark—GRPO does better on Frames (72.2 vs. 70.1) and NarrativeQA (61.0 vs. 56.0), while DAPO does better on 2WikiMultihopQA (90.5 vs. 87.0), Musique (69.0 vs. 66.0), and Qasper (58.5 vs. 56.0). On the 14B model, DAPO and GRPO are nearly identical (68.3 vs. 68.2 average). This algorithmic robustness supports the paper's claim that the progressive context scaling framework—not the choice of RL algorithm—is the primary driver of performance gains.

Limited Efficacy of SFT Alone (Table 4, SFT rows)

The warm-up SFT models show modest and inconsistent improvements over the base models. R1-Distill-Qwen-14B-SFT achieves an average of 65.0%, only +0.8 points over the base (64.2%), with losses on DocMath (-1.0) and Qasper (-2.5). R1-Distill-Qwen-32B-SFT achieves 68.7%, +3.2 points over the base (65.6%), with gains on Frames (+4.6), Musique (+4.5), and Qasper (+4.0), but no improvement on HotpotQA (+0.0). The paper presents this as evidence that SFT provides an insufficient mechanism for long-context reasoning improvement—the gains are small, fragile across benchmarks, and dwarfed by the subsequent RL gains.

However, this conclusion requires nuance. The SFT models were trained on only 5,305 examples with a 20K token context length limit, while being evaluated on benchmarks with average lengths up to ~30K and maximum lengths up to ~176K tokens. The SFT's modest gains could reflect either the inherent limitation of imitation learning for reasoning tasks (the paper's interpretation) or the mismatch between SFT training length and evaluation length—a question the paper partially addresses in the long-context SFT experiment (Figure 6), where a 10K-example long-context SFT model shows better performance (+2.6 over base on average across a subset of benchmarks) but still substantially underperforms RL-augmented models.

Gains from RL Beyond SFT (Table 4, QWENLONG-L1 rows vs. SFT rows)

The critical comparison is not SFT vs. base but SFT+RL vs. SFT alone. For the 14B model: GRPO adds +3.2 points over SFT (68.2 vs. 65.0), and DAPO adds +3.3 points (68.3 vs. 65.0). For the 32B model: GRPO adds +1.6 points over SFT (70.3 vs. 68.7), and DAPO adds +2.0 points (70.7 vs. 68.7). The RL gains are larger than the SFT gains for the 14B model (+3.2 vs. +0.8) but smaller for the 32B model (+2.0 vs. +3.2), consistent with the paper's finding that RL benefit is modulated by the quality of the SFT initialization.

The 32B model's SFT already achieves 68.7%, reducing the headroom for RL improvement compared to the 14B model's SFT at 65.0%. This diminishing returns pattern is expected—a better initialization leaves less room for improvement—but it also means the paper's claim that "RL is indispensable for attaining optimal results" (Section 4.3) is more strongly supported for the smaller model than the larger one. For the 32B model, SFT alone already matches or exceeds several strong baselines (Qwen3-32B at 67.8%, Gemini-2.0-Flash-Thinking at 65.7%), and RL provides an additional but relatively modest 2-point boost.

Comparison with State-of-the-Art Models (Table 4, Proprietary and Open-Source sections)

QWENLONG-L1-32B's 70.7 average places it in a competitive tier with several substantially larger models. Notably:

• vs. DeepSeek-R1 (72.1%): The paper's text claims QWENLONG-L1-32B outperforms DeepSeek-R1, but Table 4 shows DeepSeek-R1 at 72.1 vs. QWENLONG-L1-32B-DAPO at 70.7. This appears contradictory. The discrepancy may arise from how the paper reports its Pass@K comparison in Figure 4 (where QWENLONG-L1-14B at Pass@2 surpasses DeepSeek-R1 at Pass@1), but the main results table shows DeepSeek-R1 with the highest score among all models. The text in Section 4.1 states QWENLONG-L1-32B "exceeds the performance of... OpenAI-o3-mini" (70.7 vs. 70.4, true) and "even comparable to Claude-3.7-Sonnet-Thinking" (70.7 vs. 70.7, true), but does not directly claim to surpass DeepSeek-R1—the abstract says "surpassing OpenAI-o3-mini, Qwen3-235B-A22B, and even matches Claude-3.7-Sonnet-Thinking," which are factually supported. It's an important nuance: QWENLONG-L1-32B is competitive with the strongest models but does not dominate them.

• vs. Qwen3-235B-A22B (70.6%): QWENLONG-L1-32B (70.7) edges out this model that has ~7× more parameters, demonstrating that targeted long-context RL can compensate for scale.

• vs. OpenAI-o1-preview (72.9%): QWENLONG-L1-32B trails by 2.2 points on average, and the gap is particularly large on NarrativeQA (61.0 vs. 68.0) and Musique (69.0 vs. 69.0, tied, but o1-preview leads on Frames 80.8 vs. 70.1). OpenAI-o1-preview was not specifically trained for long-context tasks, making this comparison interesting but not dispositive.

• vs. Gemini-2.0-Flash-Thinking (65.7%): QWENLONG-L1-14B (68.3) already surpasses this model by 2.6 points, demonstrating that even the smaller QWENLONG-L1 variant competes favorably with some proprietary thinking models.

Test-Time Scaling Behavior (Figure 4)

The paper evaluates Pass@K for K = 1, 2, 4, 8, 16 using QWENLONG-L1-14B (the specific variant is not explicitly stated in Figure 4, but the surrounding text discusses "QWENLONG-L1-14B with different sample numbers"). The key finding:

• Pass@2 of QWENLONG-L1-14B achieves 73.7 averaged across all benchmarks, surpassing DeepSeek-R1's 72.1 (Pass@1, single sample) and OpenAI-o1-preview's 72.9 (Pass@1). This is reported in the text as demonstrating that QWENLONG-L1-14B "surpasses DeepSeek-R1 with a small sampling number."

• Pass@16 continues to improve, suggesting the model's output distribution contains high-quality solutions that are not reliably found with a single sample—the gap between Pass@1 and Pass@16 is substantial. The paper notes that "the significant gap between Pass@K and Pass@1 metrics indicates further potential for RL training to better bridge the transition from diverse exploration to optimal exploitation" (Section 4.1, final paragraph). This is an honest acknowledgment that the model's Pass@1—the metric used for the main comparison in Table 4—understates its potential, and that better training strategies or test-time selection mechanisms could close the exploration-exploitation gap.

A critical limitation: Figure 4 reports Pass@K curves averaged across all seven benchmarks, not broken out by benchmark. The paper does not show whether the Pass@K improvement is uniform (all benchmarks benefit similarly from sampling) or concentrated in specific benchmarks. Given the heterogeneity in benchmark-level results (Table 4), this aggregation may mask important variation—for example, the model might be near ceiling on 2WikiMultihopQA (90.5% Pass@1) with little room for Pass@K improvement, while showing dramatic gains on DocMath.

Ablation Studies and Robustness Checks

The ablation studies in Section 4.2 (Figure 5) test the three components of the progressive context scaling framework. All ablations use the 14B model variant and report accuracy averaged across the seven benchmarks.

Warm-up SFT (Figure 5a, comparing bars with and without SFT): Adding warm-up SFT before RL improves performance across all configurations. For GRPO with phased RL and retrospective sampling: SFT + GRPO + phased RL + RS achieves approximately 68.2 (from Table 4, QWENLONG-L1-14B-GRPO), while the same without SFT (GRPO + phased RL + RS) achieves approximately 64–65 (estimated from Figure 5a bar chart—exact numbers are not quoted precisely, but the difference is visually clear). The mechanism is revealed in Figure 5b: SFT leads to "higher initial reward and lower gradient norm," indicating more stable optimization from the start of RL training. This is consistent with SFT establishing format compliance and basic grounding before RL begins.

Curriculum-guided phased RL vs. single-stage RL (Figure 5a, comparing "RL" and "Phased RL" bars): Phased RL outperforms single-stage RL across all configurations. For SFT + GRPO: phased RL achieves approximately 68.2, while single-stage RL achieves approximately 65–66 (estimated from Figure 5a). The gap is consistent but moderately sized—roughly 2–3 points. Figure 5c provides the mechanistic explanation: single-stage RL exhibits "fluctuating KL divergence and entropy collapse," while phased RL maintains more stable KL divergence and higher entropy. This directly addresses the two failure modes identified in Figure 2 (delayed reward convergence from entropy reduction, unstable optimization from KL divergence spikes). A notable interaction effect: the improvement from phased RL is less pronounced when SFT is used (the gap between single-stage and phased RL narrows for SFT-initialized models), suggesting that SFT "partially compensates for curriculum design" (Section 4.2). This is an interesting finding: SFT and phased RL appear to address overlapping aspects of the instability problem, with SFT providing a strong enough initialization that the curriculum becomes somewhat less critical.

Difficulty-aware retrospective sampling (Figure 5a, comparing "Phased RL" and "Phased RL + RS" bars): Adding retrospective sampling on top of phased RL provides a further performance boost of approximately 1–2 points (estimated from Figure 5a—the paper does not quote exact deltas). Figure 5d shows that the retained hard examples exhibit "significantly lower reward and higher policy entropy" during Phase II training, confirming that retrospective sampling maintains exploration pressure for difficult reasoning problems as the curriculum progresses to longer contexts. Without retrospective sampling, Phase II training would consist entirely of longer but not necessarily harder examples, potentially allowing the policy to converge prematurely on strategies that work for long-but-easy problems while neglecting genuinely hard reasoning challenges.

Negative or null results: The paper does not report any ablation where a component hurts performance—all three components (SFT, phased RL, retrospective sampling) show additive positive effects. This could indicate that each component addresses a genuinely distinct aspect of the training instability, or it could reflect the limited scope of the ablation (testing only within the specific configuration space of 14B model, GRPO/DAPO, 20K → 60K curriculum). A valuable but absent ablation would be testing whether the difficulty-aware retrospective sampling helps when applied to single-stage RL (without curriculum)—this would disentangle whether retrospective sampling's benefit comes from maintaining hard examples specifically across a phase transition, or whether it simply provides a better training distribution in any RL setting.

SFT-RL tradeoff (Figure 6, Section 4.3): The paper tests three initialization strategies with the 14B model: base model → RL, short-context SFT → RL, and long-context SFT (10K examples, full context length) → RL. The results: long-context SFT achieves the best standalone performance (68.2 estimated from text: base at 64.2, +2.6 from long-context SFT = 66.8, but the text says "surpasses both the base model by 2.6 points and the short-context SFT model by 2.1 points"—the exact numbers are difficult to reconstruct without Figure 6 values). Critically, when RL is applied: long-context SFT + RL achieves only +0.3 additional points (to approximately 67.1–67.4), while short-context SFT + RL achieves +3.2 additional points (to 68.2, matching QWENLONG-L1-14B-GRPO from Table 4). The final performance of short-context SFT + RL exceeds that of long-context SFT + RL, despite the long-context SFT model starting from a better baseline. This is the paper's most surprising result and its strongest evidence for the claim that RL should be prioritized over SFT.

Reasoning behavior emergence (Figure 7, Section 4.3): The paper uses DeepSeek-V3 to automatically annotate the frequency of four reasoning behaviors (grounding, subgoal setting, backtracking, verification) in model outputs over the course of SFT and RL training. The key findings: (1) Both SFT and RL increase the frequency of all four behaviors. (2) During RL, behavior frequency increases progressively and correlates with performance gains. (3) During SFT, behavior frequency increases but does not correlate with performance gains—"these adjustments fail to transform into performance improvements" (Section 4.3). The paper interprets this as evidence that SFT learns superficial reasoning patterns (the form of reasoning without the function), while RL refines the output distribution so that these behaviors, when they occur, are more likely to lead to correct answers. This is a provocative finding but the methodology has limitations: the behavior annotation uses an LLM judge (DeepSeek-V3), and the paper does not report inter-annotator agreement, human validation, or error analysis of the behavior classification. It is possible that SFT increases the apparent frequency of reasoning behaviors (the model learns to produce text that looks like reasoning to the judge) without actually performing effective reasoning, while RL genuinely improves reasoning quality—but this alternative interpretation is consistent with the paper's own interpretation and would produce the same observed pattern.

Algorithm comparison (Table 4, GRPO vs. DAPO rows): The paper evaluates both GRPO and DAPO at both model scales. Results: 14B—GRPO 68.2, DAPO 68.3 (near-identical); 32B—GRPO 70.3, DAPO 70.7 (DAPO slightly ahead). The DAPO advantage on the 32B model is small (0.4 points) and concentrated in specific benchmarks (2WikiMultihopQA: 90.5 vs. 87.0; Musique: 69.0 vs. 66.0; Qasper: 58.5 vs. 56.0), while GRPO leads on Frames (72.2 vs. 70.1) and NarrativeQA (61.0 vs. 56.0). This mixed pattern suggests neither algorithm is universally superior, and the choice may depend on the specific benchmark characteristics. DAPO's additional mechanisms (asymmetric clipping, dynamic sampling, length penalty) appear to help on certain reasoning types but not others—understanding this heterogeneity would require a deeper analysis of which benchmarks benefit from which DAPO components.

Critical Assessment

The paper makes three central claims that the experiments should support, and a fourth implicit claim that emerges from the framing.

Claim: "QWENLONG-L1 achieves leading performance among state-of-the-art proprietary LRMs, with QWENLONG-L1-32B matching Claude-3.7-Sonnet-Thinking." The evidence supports this, but with important scope boundaries. Table 4 shows QWENLONG-L1-32B-DAPO at 70.7, tied with Claude-3.7-Sonnet-Thinking (70.7) and ahead of OpenAI-o3-mini (70.4) and Qwen3-235B-A22B (70.6). However, QWENLONG-L1-32B trails DeepSeek-R1 (72.1) and OpenAI-o1-preview (72.9), both of which are "state-of-the-art proprietary LRMs" that should be included in any claim of "leading performance." The paper's text carefully avoids claiming to outperform DeepSeek-R1 (the abstract says "surpassing OpenAI-o3-mini, Qwen3-235B-A22B" and "comparable to Claude-3.7-Sonnet-Thinking," all of which are true), but the broader framing of "leading performance" should be qualified: QWENLONG-L1 leads among models of its scale class and in its specific long-context DocQA niche, but does not uniformly lead across all LRMs. Additionally, the comparison against proprietary models is not controlled—the baselines are evaluated with truncated context windows (≤50K tokens, Section 3.3) and no optimization for the long-context setting, while QWENLONG-L1 has been specifically trained for this task distribution. This is a confound: the comparison measures (proprietary model with generic setup) vs. (QWENLONG-L1 with task-specific training), not (best possible proprietary model) vs. (QWENLONG-L1). A fairer comparison would give the proprietary models the same maximum context length and, ideally, the same prompt template.

Claim: "SFT alone fails to achieve substantial improvements, while RL provides significant gains." This is supported but requires careful interpretation. Table 4 shows SFT gains of +0.8 (14B) and +3.2 (32B) points, which are indeed smaller than the RL gains of +4.0–4.1 (14B) and +4.7–5.1 (32B). However, the SFT models were trained on only 5,305 examples with a 20K context cap, and the 32B SFT model's 68.7 already surpasses Qwen3-32B (67.8) and Gemini-2.0-Flash-Thinking (65.7). Calling SFT's gains "limited" is accurate relative to RL, but the absolute performance of the SFT models is competitive with several strong baselines—the SFT alone produces a capable long-context reasoner, just not an optimal one. The stronger version of this claim—that SFT is inherently limited for long-context reasoning—is supported by the long-context SFT experiment (Figure 6), where even 10K examples of full-length SFT data plateau at 0.3 points of RL improvement. This provides converging evidence that the limitation is not just about the amount of SFT data or its context length coverage, but about a fundamental ceiling on what imitation learning can achieve for reasoning tasks.

Claim: "Progressive context scaling is essential for stable long-context RL training." The ablation study (Figure 5a) demonstrates that removing progressive scaling (single-stage RL) substantially reduces performance. The mechanism evidence (Figures 5c, 5d) provides a plausible causal chain: single-stage RL suffers from entropy collapse and KL instability → phased RL prevents these → retrospective sampling maintains exploration on hard examples → performance improves. This evidence is strong within the paper's experimental scope, but the "essential" claim is not fully tested. The paper tests only two configurations of the curriculum (20K → 60K) and only one curriculum structure (two discrete phases). It does not test whether alternative stabilization methods—different learning rates, different clipping thresholds, reward shaping, or reference model update frequency—could achieve similar stability without a curriculum. It does not test whether a finer-grained curriculum (3+ phases) or different length thresholds would improve or degrade performance. The curriculum is tested only in conjunction with group-relative RL algorithms; it's possible that PPO with a value network, despite the computational cost, might be more robust to input length heterogeneity and reduce the need for curriculum. The paper demonstrates that its specific progressive scaling approach works, not that progressive scaling is the only way to stabilize long-context RL.

Implicit claim: "The RL training framework generalizes to contexts longer than those seen during training." The paper's training uses a maximum of 60K token inputs (Phase II), but evaluation uses up to 120K tokens, and benchmarks have maximum lengths up to 176K tokens (DocMath). The strong evaluation results suggest some length generalization, but the paper never isolates this effect. It does not report performance broken out by input length bucket—for example, do QWENLONG-L1 models maintain accuracy on the longest examples in DocMath (100K+ tokens), or does performance degrade beyond the 60K training maximum? Without this analysis, we cannot tell whether the model has genuinely learned length-generalizable grounding strategies, or whether the strong average performance is driven by the majority of benchmark examples that fall within or near the training length distribution. Table 3 shows average benchmark lengths of 5K–30K tokens, well within the 60K training cap; only the maximum lengths extend beyond, and those may represent a small fraction of the test set. This is a significant gap: demonstrating length generalization would require reporting performance stratified by input length.

Missing experiments that would strengthen the paper:

• Stratification by input length. Reporting accuracy for short (≤20K), medium (20K–60K), and long (60K–120K+) examples separately would reveal whether progressive scaling achieves genuine length generalization or mainly improves performance within the training distribution.

• Ablation of the phase transition design. Testing whether performance degrades if Phase II uses the same input length distribution as Phase I but with different examples (i.e., the curriculum is about training duration, not length), or whether increasing context length without reducing the number of shorter examples (mixed-length training rather than length-filtered phases) achieves similar stability.

• Ablation of the retrospective sampling difficulty threshold. Testing whether carrying forward only the hardest examples (average reward zero) is optimal, versus a softer threshold, versus random sampling of Phase I examples, versus no retrospective sampling.

• Statistical testing. With benchmark test sets of 200–824 questions, differences of 1–2 points between models are within the range where sampling noise matters. Confidence intervals or bootstrap estimates would clarify whether QWENLONG-L1's edge over, e.g., Qwen3-235B-A22B (70.7 vs. 70.6) is meaningful.

• Comparison with long-context-trained non-reasoning models. The paper compares against reasoning models (LRMs) but not against models specifically optimized for long-context QA without explicit reasoning (e.g., LongLoRA-based models, models with specialized retrieval mechanisms). This would help isolate whether the benefit comes from reasoning RL specifically, or from any form of long-context training.

• Analysis of training cost. The paper reports hardware (32×A100) and hyperparameters but does not report total training time, total FLOPs, or compare the computational cost of QWENLONG-L1 training against the cost of training the baseline models. This makes it difficult to assess whether the performance gains are cost-effective relative to simply training a larger model or doing more SFT.

Where the claims hold conditionally:

• The advantage over proprietary models holds for the specific DocQA benchmarks and evaluation protocol (zero-shot, temperature 0.7, max 120K input), but would likely weaken if proprietary models were evaluated with their full context windows (many support 200K+ tokens) and with prompts optimized for long-context reasoning.

• The SFT-RL tradeoff finding (Figure 6) holds for the tested configuration (14B model, 10K SFT examples, full-length context, single-stage RL), but may not generalize to different amounts of SFT data, different SFT quality, or different RL algorithms. The paper's recommendation to "prioritize RL over SFT" should be understood as "for this specific model family and task, extensive SFT on long-context data is counterproductive," not as a universal principle.

• The progressive scaling benefit holds for the tested two-phase 20K → 60K curriculum with group-relative RL algorithms; it is unclear whether the benefit would persist with more phases, different length thresholds, or alternative RL algorithms that might be more robust to input length variance.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Efficiency Claims

The assumption or constraint. The progressive context scaling framework relies on difficulty-aware retrospective sampling, which requires quantifying each training example's difficulty as the inverse mean reward of groups of outputs generated by the base model (Equation 10: diffs(x,c) = 1 / mean({r_i})). Section 3.2 states that the framework "incorporate(s) difficulty-aware retrospective sampling to maintain the most difficult samples with an average accuracy of zero from phase I to II." Computing these difficulty scores requires generating G = 8 complete outputs for every example in the Phase I training set (1,591 examples in DocQA-RL-1.6K) using the base model before RL training begins—a cost of approximately 12,728 full-length inference passes. The paper does not account for this cost in any training budget, total FLOPs calculation, or wall-clock time estimate.

The consequence. In a real deployment pipeline, difficulty estimation adds a significant computational overhead that is entirely separate from and prior to the RL training itself. For the specific configuration reported (14B model, 1.6K training examples, 8 rollouts per example), this represents roughly 1.6K × 8 × (avg output length) tokens of inference before a single RL update occurs. The relative cost depends on the average output length (not reported for the base model on the RL dataset), but even conservative estimates place it in the range of 10²-10³ GPU-hours that are not reflected in the paper's training cost. More critically, the difficulty scores are computed once from the base model and remain fixed throughout training—they do not adapt as the policy improves. This means the retrospective sampling retains examples that were difficult for the untrained base model, which may not be the same examples that remain difficult or provide useful exploration signal later in training. If the base model's difficulty ranking does not align with the policy's learning trajectory, the retrospective sampling could retain uninformative examples while discarding examples that would have provided more valuable gradient signal.

What evidence exists in the paper. The paper does not report the computational cost of difficulty estimation, does not include it in any budget calculation, and does not ablate whether computing difficulty from the current policy at each phase transition (rather than from the fixed base model) would change which examples are retained or affect final performance. Section 3.2 states that the training uses 32×A100-80G GPUs and provides learning rates, batch sizes, and rollout numbers, but the difficulty estimation step is not mentioned in the training details section and its cost is never quantified. The ablation in Figure 5d shows that retained examples maintain lower reward and higher entropy during Phase II training, confirming they remain challenging, but this is measured after the retention decision has been made using the base model's fixed difficulty scores—it cannot tell us whether examples that were not retained might have provided even better exploration signal.

Mitigation status. Not addressed. The paper does not discuss the cost of difficulty estimation, propose cheaper alternatives (e.g., using fewer rollouts, using a smaller model for difficulty scoring, or using heuristic length-based or perplexity-based difficulty proxies), or analyze whether the specific choice of base-model-based fixed difficulty is optimal. The finding in Figure 5d that retained examples maintain high entropy during Phase II provides partial validation of the approach (the retained examples are indeed still hard), but does not address whether alternative difficulty estimation strategies would be more cost-effective or yield better retention sets.

Evaluation Is Restricted to a Single Task Family (Document QA) and a Single Model Family (R1-Distill-Qwen)

The assumption or constraint. All experiments use the MATH of long-context reasoning: document question answering over seven DocQA benchmarks. The base models are exclusively R1-Distill-Qwen-14B and R1-Distill-Qwen-32B—distilled from DeepSeek-R1 and inheriting Qwen's pretraining architecture and data distribution. Section 4 states: "In our experiments, we employ document question answering (DocQA) as our primary evaluation task for long-context reasoning capabilities, as it inherently requires both contextual grounding and multi-step reasoning." The paper frames this choice as providing a representative testbed, but does not evaluate on other long-context reasoning domains such as long-form code generation (reasoning over multi-file codebases), legal document analysis (statutory interpretation across lengthy case law), scientific literature review (synthesis across dozens of papers), or long-context dialogue and debate.

The consequence. The paper's central claim—that progressive context scaling enables stable long-context reasoning RL—is demonstrated for exactly one reasoning format (extractive + computational question answering over provided documents) using exactly one model family. DocQA tasks share specific structural properties that may be more amenable to the approach than other long-context reasoning domains: (1) the answer is typically a short extractive span or a computed value, making reward computation via hybrid exact-match + LLM-judge straightforward; (2) the reasoning process is predominantly grounding (finding and synthesizing information from the context) rather than generating novel content, meaning the model can succeed by developing strong retrieval-and-synthesis strategies without necessarily improving its ability to produce original long-form reasoning; (3) the evaluation benchmarks all follow similar formats (question, context, answer) with answer extraction templates that align with the training prompt structure. Whether progressive context scaling transfers to, say, long-context mathematics where the model must generate multi-page proofs referencing lemmas distributed across a textbook, or to long-context code generation where the model must maintain consistency across multiple files, is completely untested. Similarly, the R1-Distill-Qwen models were specifically trained (via distillation from DeepSeek-R1) on short-context mathematics, coding, and scientific reasoning—they may exhibit reasoning behaviors or output distributions that interact favorably with progressive context scaling in ways that other model families (e.g., LLaMA-based reasoning models, non-distilled RL-trained models) would not replicate.

What evidence exists in the paper. Table 4 shows benchmark-level heterogeneity even within DocQA: QWENLONG-L1-32B-DAPO gains range from +2.0 points on NarrativeQA to +8.5 points on Qasper relative to the base model, suggesting the framework's effectiveness varies substantially even within the single task family. The case studies in Section 6 show examples from DocMath (financial report reasoning) but not from the other six benchmarks. The paper does not report any experiment using a different base model family, a different pretraining corpus, or a different distillation source, so the interaction between model architecture/pretraining and progressive context scaling is entirely unexplored. The ablation studies (Figures 5, 6) are exclusively on the 14B variant, meaning even within the paper's own model family, we lack ablation results at 32B scale to confirm that the same mechanisms (entropy stabilization, KL divergence reduction from phased RL) operate similarly at larger model sizes.

Mitigation status. Not addressed. The paper's contributions list (Section 1) claims to provide "a fundamental recipe and practice for long-context reasoning optimization," but the generality of this recipe is asserted rather than demonstrated. The paper does not discuss domain-specific features of DocQA that might limit transfer, does not propose experiments on other long-context reasoning domains, and does not test alternative base models. Future work on "scaling real-world tasks, like automated scientific research and long video analysis" is mentioned in Section 5, which implicitly acknowledges the current scope limitation, but no near-term validation on related domains (e.g., long-context code reasoning, which shares many structural properties with DocQA) is reported.

Training and Evaluation Use Different Judge Models with Unquantified Distribution Shift

The assumption or constraint. The hybrid reward mechanism (Section 2.4) uses Qwen2.5-1.5B-Instruct (temperature 0) as the LLM judge during RL training—a small, efficient model chosen to enable online reward computation without becoming the computational bottleneck. For evaluation (Section 3.3), the paper switches to DeepSeek-V3 with temperature 0.0: "We use DeepSeek-V3 as the judge model with a temperature of 0.0 to provide a reliable evaluation." This creates a systematic mismatch between the reward signal the policy optimizes during training and the correctness criterion used to evaluate it at test time.

The consequence. The training reward function r_phi(x, y) = max(r_rule(y), r_LLM(x, y)) depends on the Qwen2.5-1.5B judge's assessment of semantic equivalence. If this small judge systematically differs from DeepSeek-V3—for example, by being more lenient (marking more answers as correct) or more strict in specific domains—the policy will be optimized for a proxy objective that does not perfectly align with the evaluation metric. The consequence depends on the direction of the mismatch:

• If Qwen2.5-1.5B is more lenient than DeepSeek-V3: the policy receives positive reward for answers that the evaluator would mark incorrect, reinforcing strategies that produce superficially plausible but incorrectly-judged-as-correct answers. The policy's Pass@1 under DeepSeek-V3 evaluation would then be lower than what the training reward curve suggests—the policy has been optimized for an easier judge.
• If Qwen2.5-1.5B is more strict than DeepSeek-V3: the policy receives negative reward for answers that the evaluator would actually accept, penalizing correct reasoning expressed in forms the small judge doesn't recognize. This would suppress exploration of valid but non-canonical answer formats, potentially reducing the diversity of correct solutions the policy discovers.

The paper provides no analysis of agreement rates, systematic biases, or domain-specific disagreement patterns between the two judges. Without this, we cannot determine whether the strong evaluation results reflect genuine reasoning improvement or partial reward hacking of the small-judge signal that happens to also satisfy the larger judge for the specific benchmark answers. The max operator in the training reward (Equation 14) partially mitigates this: answers that match the gold string exactly bypass the LLM judge entirely, and the rule-based component provides an anchor of ground-truth correctness. However, for the substantial fraction of evaluation examples where exact match fails and the LLM judge is the sole determinant of correctness, the judge mismatch directly affects the alignment between training objective and evaluation metric.

What evidence exists in the paper. The paper does not report judge agreement rates, does not analyze cases where Qwen2.5-1.5B and DeepSeek-V3 disagree, and does not provide any characterization of the small judge's reliability relative to the large judge. Section 2.4 justifies the small judge by stating it "enables efficient reward computation during online RL training while maintaining evaluation reliability," but "evaluation reliability" is never quantified. The fact that DeepSeek-V3 is used for evaluation rather than Qwen2.5-1.5B implicitly concedes that the small judge is not sufficiently reliable for evaluation—but the training process has already been shaped by its judgments.

Mitigation status. Not addressed. The paper does not propose or test alternatives that would reduce the judge mismatch: using the same judge for training and evaluation, training with a larger judge but amortizing cost via offline reward precomputation, or analyzing the sensitivity of final performance to judge quality. The choice is presented as a practical compromise without analysis of its consequences.

The Hardest Evaluation Examples Show No Measurable Improvement, Establishing a Capability Ceiling

The assumption or constraint. The progressive context scaling framework operates under the assumption that the base model possesses sufficient underlying capability to solve long-context reasoning problems if given stable RL training—the framework improves how the model reasons, but does not teach entirely new capabilities. This assumption is validated for easy-to-medium examples (where the model's initial pass@1 is non-zero) but fails catastrophically for the hardest examples. While the paper does not provide a formal difficulty-bin analysis (as in the compute-optimal test-time scaling paper), the training dynamics reveal this ceiling: the difficulty-aware retrospective sampling specifically retains examples with "an average accuracy of zero from phase I to II" (Section 3.2), indicating that a substantial subset of training examples remain at zero reward even after Phase I RL training. These are examples where the model never produces a correct answer across 8 rollouts—not just at initialization, but after an entire phase of targeted RL optimization.

The consequence. For problems where the base model's capability is fundamentally insufficient—where no amount of grounding improvement or reasoning refinement can bridge the gap to correctness—QWENLONG-L1 provides no benefit. The retained hard examples in Phase II maintain "higher policy entropy" (Figure 5d), indicating continued exploration, but the paper does not report whether any of these zero-reward examples eventually achieve non-zero reward during Phase II. If they do not, the retrospective sampling is maintaining examples that consume training compute without providing positive learning signal—they contribute only through their effect on the group-relative advantage normalization (by providing low-reward baselines that make high-reward outputs' advantages more positive). This is a legitimate use of negative examples, but it means the framework's compute is partially spent on problems it cannot solve, which may be suboptimal compared to a strategy that discards truly impossible examples and focuses training budget on examples within the model's capability frontier.

More importantly, this ceiling has direct implications for deployment: QWENLONG-L1 will systematically fail on problems that exceed the base model's fundamental reasoning capacity, regardless of how much RL training it receives or how stable that training is. The framework amplifies existing capability but does not create it. This is especially relevant for the "deep research" applications the paper cites as motivation (Section 1)—real-world document analysis often involves genuinely novel reasoning challenges that may exceed even strong base models' capabilities.

What evidence exists in the paper. The paper reports that retrospective sampling retains examples with "average accuracy of zero" (Section 3.2), confirming the existence of such examples. Figure 5d shows these examples exhibit higher entropy during Phase II—the model remains uncertain—but does not report whether their reward ever increases above zero. The benchmark-level results in Table 4 show substantial variance in improvement: +8.5 points on Qasper vs. +2.0 on NarrativeQA for the 32B model. This could indicate that some benchmarks contain a higher proportion of "impossible" examples where the base model's capability is insufficient, but the paper does not analyze this. The Pass@K analysis (Figure 4) shows that even with 16 samples (Pass@16), accuracy does not approach 100%, meaning some questions remain unsolved at any sampling budget—consistent with a capability ceiling, though sampling budget is not the same as training budget.

Mitigation status. Partially acknowledged. Section 5's future work mentions "scaling real-world tasks... will provide appropriate environments to enhance long-context comprehension and decision-making capabilities," which implicitly recognizes that current training environments may not be rich enough to teach all necessary capabilities. However, the paper does not directly discuss the capability ceiling, does not analyze what characterizes problems that remain at zero reward, and does not propose mechanisms for identifying and routing such problems to alternative solution strategies (e.g., larger models, human review). The difficulty-aware retrospective sampling retains hard examples but does not solve them—it is a mechanism for maintaining exploration pressure, not for transcending the capability frontier.

Sequential Dependence of the Three-Stage Pipeline Creates a Brittle and Costly Training Recipe

The assumption or constraint. QWENLONG-L1's training pipeline consists of three sequential stages—warm-up SFT, Phase I RL, Phase II RL—where each stage depends on the output of the previous stage, and hyperparameters from early stages interact with downstream performance in ways that are not independently tunable. The SFT stage (Section 2.3) must produce a policy that is neither too weak (failing to generate any correct outputs during RL, producing zero reward variance and no learning signal) nor too strong (overfitting to the SFT distribution and constraining RL exploration, as demonstrated in Figure 6). The Phase I RL must train long enough to establish grounding capabilities but not so long that entropy collapses before Phase II begins. The choice of 20K tokens for Phase I and 60K tokens for Phase II is a single tested pair; whether a 10K → 40K curriculum, a 30K → 90K curriculum, or a three-phase 20K → 40K → 60K curriculum would perform better (or worse) is unknown.

The consequence. The training recipe is fragile in two senses. First, reproducibility: a practitioner attempting to apply QWENLONG-L1 to a new base model or task domain cannot simply adopt the paper's hyperparameters and expect similar results. The optimal SFT duration, Phase I length, Phase II length, and retrospective sampling threshold likely depend on the base model's initial capability, the target task's difficulty distribution, and the interaction between these factors—none of which are characterized in the paper. If a practitioner trains SFT for too long (as in the Figure 6 long-context SFT experiment), RL gains collapse from ~3 points to ~0.3 points—a 10× reduction in RL effectiveness that would be discovered only after completing the full RL training pipeline, wasting substantial compute.

Second, cost of hyperparameter search: each configuration of the pipeline requires running all three stages to evaluate, because the interaction effects are sequential—you cannot evaluate Phase II RL independently of the Phase I RL and SFT that preceded it. Testing N different SFT configurations × M different Phase I configurations × K different Phase II configurations would require N×M×K full training runs, each taking 32×A100 GPUs for the duration of SFT (3 epochs on 5.3K examples) plus two phases of RL (of unspecified duration). The paper's ablation studies (Figure 5) test only binary presence/absence of each component, not their hyperparameter sensitivity. The SFT sensitivity analysis (Figure 6) tests exactly two SFT variants (short-context and long-context). The Phase transition is tested for exactly one pair of thresholds (20K → 60K). No learning rate sensitivity, batch size sensitivity, rollout number sensitivity, or phase duration sensitivity is reported.

This fragility is particularly problematic because the paper's key finding—that SFT can be too effective and trap models in local optima—means that the practitioner cannot simply default to "more SFT is better" and must instead tune SFT to a narrow window where it provides sufficient initialization without over-constraining RL. Finding this window for a new model/task combination without the kind of extensive experimentation the paper itself performed would be challenging.

What evidence exists in the paper. The Figure 6 experiment directly demonstrates the brittleness: changing from short-context SFT to long-context SFT reduces subsequent RL gains from 3.2 points to 0.3 points, a 10.7× reduction. The ablation in Figure 5a shows that removing any single component (SFT, phased RL, or retrospective sampling) reduces performance, but does not show how performance varies with the degree of each component. The paper does not report any sensitivity analysis for the key hyperparameters: Phase I threshold (20K), Phase II threshold (60K), SFT epochs (3), SFT learning rate (5e-6), RL learning rate (2e-6), rollout number (8), or the difficulty threshold for retrospective sampling (average accuracy of zero).

Mitigation status. Not addressed. The paper provides a single training recipe with fixed hyperparameters and demonstrates it works for one model family on one task family. Whether this recipe transfers to new settings—and how much tuning is required to make it work—is an open question. The paper's presentation as a "framework" and "fundamental recipe" (Section 1, contributions) implies generality that has not been empirically validated.

Test-Time Compute Scaling Analysis Confounds Training Improvement with Sampling Diversity

The assumption or constraint. The paper's test-time scaling analysis (Section 4.1, Figure 4) evaluates Pass@K for QWENLONG-L1-14B by generating 16 candidate outputs per question and computing the fraction of questions where at least one of the top-K outputs is correct. The paper compares QWENLONG-L1-14B's Pass@2 (73.7) against DeepSeek-R1's Pass@1 (72.1) and OpenAI-o1-preview's Pass@1 (72.9), concluding that QWENLONG-L1-14B "surpasses DeepSeek-R1 and OpenAI-o1-preview with a small sampling number." This comparison uses Pass@K for QWENLONG-L1 but Pass@1 for the baselines—an asymmetric evaluation that conflates the model's training-induced quality improvement with the benefits of sampling diversity that were not applied to the baseline models.

The consequence. The Pass@K comparison does not isolate QWENLONG-L1's training benefit. Any model—including the baselines—would improve with multiple samples, because Pass@K is monotonically non-decreasing in K. Without evaluating Pass@K for DeepSeek-R1, OpenAI-o1-preview, and other baselines, the paper cannot distinguish between two scenarios: (1) QWENLONG-L1's RL training genuinely produces a better output distribution (higher Pass@1 and higher Pass@K for all K), or (2) QWENLONG-L1's training produces a more diverse output distribution (similar Pass@1 but faster Pass@K growth), which improves best-of-K performance but for a different reason. If the baselines also show strong Pass@K scaling—which is common for reasoning models that generate diverse chain-of-thought trajectories—the claimed advantage of QWENLONG-L1 at Pass@2 might disappear or reverse at equal K. A particularly important missing comparison: the base model R1-Distill-Qwen-14B's Pass@K curve. Without this, we cannot determine how much of QWENLONG-L1's Pass@K improvement comes from RL training versus simply from the base model's inherent sampling diversity.

What evidence exists in the paper. Figure 4 shows QWENLONG-L1-14B's Pass@K curve increasing from approximately 68–69 at Pass@1 to approximately 80 at Pass@16. The paper does not report Pass@K for any baseline model. The comparison text in Section 4.1 states: "QWENLONG-L1-14B demonstrates remarkable gains, even surpassing DeepSeek-R1 and OpenAI-o1-preview with a small sample size. Specifically, it achieves an average Pass@2 rate of 73.7 across all benchmarks, outperforming both DeepSeek-R1's 72.1 and OpenAI-o1-views's 72.9." The implied comparison is Pass@2 (QWENLONG-L1) vs. Pass@1 (baselines), but this is not made explicit in the text—it requires reading the experiment description to understand that baselines were evaluated with single outputs. The paper also does not report Pass@1 for QWENLONG-L1-14B separately from the Table 4 results (which use the same evaluation protocol and thus represent Pass@1). From Table 4, QWENLONG-L1-14B-DAPO achieves 68.3 average, which is the model's actual Pass@1—this is substantially lower than DeepSeek-R1's 72.1 Pass@1, meaning QWENLONG-L1-14B at Pass@1 does not surpass DeepSeek-R1. The claimed advantage exists only at Pass@2 versus Pass@1.

Mitigation status. Not addressed. The paper does not report Pass@K for baselines, does not control for sampling budget in its comparisons, and does not discuss the confound. The final paragraph of Section 4.1 acknowledges that "the significant gap between Pass@K and Pass@1 metrics indicates further potential for RL training to better bridge the transition from diverse exploration to optimal exploitation," which correctly identifies the exploration-exploitation gap but does not address the fairness of comparing models at different K values. The paper's test-time scaling analysis would be substantially strengthened by reporting Pass@K curves for at least the base model (R1-Distill-Qwen-14B) and the strongest open-source baseline (DeepSeek-R1), enabling a direct assessment of how much QWENLONG-L1 improves the output distribution's quality versus its diversity.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes how the field should think about the relationship between supervised fine-tuning and reinforcement learning for reasoning models, and it establishes long-context reasoning RL as a distinct optimization regime rather than a trivial extension of short-context methods. The magnitude of this shift differs across audiences: for practitioners building long-context reasoning systems, it is a methodological reframing that replaces the default assumption of "apply the same RL recipe that worked for math and coding" with an explicit awareness of context-length-dependent training dynamics. For researchers studying RL for language models, it is a diagnostic contribution that introduces input length heterogeneity as a first-class variable in stability analysis. For the broader field, it is an incremental but practically significant advance that bridges a recognized gap between short-context reasoning RL (where extensive prior work exists) and long-context deployment scenarios (where such work was absent).

The SFT-RL relationship is fundamentally reconfigured. Prior to this work, the dominant assumption—implicit in essentially all reasoning model training pipelines from DeepSeek-R1 [11] onward—was that more and better SFT data provides a uniformly better initialization, and RL then provides additional optimization on top. The paper's Figure 6 experiment inverts this intuition: a long-context SFT model trained on 10K examples at full context length achieves better standalone performance than a short-context SFT model (+2.1 points), yet when RL is applied, it gains only 0.3 points compared to the short-context model's 3.2 points. The model that received more supervised training on exactly the target distribution ends up worse after RL than the model that received less. This is not a small effect—it is a ~10× difference in RL efficiency—and it implies that the standard "SFT then RL" pipeline has a sweet spot that practitioners must calibrate rather than maximize. The paper's behavior analysis (Figure 7) provides the mechanistic explanation: SFT increases the frequency of reasoning behaviors but "these adjustments fail to transform into performance improvements," while RL refines the output distribution so that these behaviors, when they occur, are more likely to lead to correct answers. SFT teaches the form of reasoning; RL teaches its function. Too much SFT solidifies the form in ways that constrain RL's ability to optimize the function.

Long-context reasoning RL is established as a distinct paradigm. The paper's preliminary diagnostic (Figure 2) does something the field had not done: systematically compare training dynamics between short-context and long-context RL under otherwise identical conditions. The result—entropy collapse and KL divergence instability emerging specifically from long-context inputs—provides a causal explanation for why naive application of short-context RL methods fails. This reframes the problem from "we don't have good long-context reasoning models" (a capability gap) to "long-context inputs produce specific, identifiable pathologies in RL training dynamics" (a mechanistic gap with defined intervention points). Future work on long-context reasoning RL can now target these specific failure modes—entropy preservation and KL stabilization—rather than treating the problem as a black-box optimization challenge. The paper's identification of these mechanisms also makes negative results more interpretable: if a new long-context RL method fails, researchers can examine entropy and KL trajectories to diagnose whether the failure matches the patterns identified in Figure 2.

The training-inference context-length gap is partially bridged—but the generalization mechanism remains opaque. The paper demonstrates that a model trained with a maximum context length of 60K tokens (Phase II) can perform strongly on benchmarks with examples up to 176K tokens (DocMath max length, Table 3). This suggests some form of length generalization, but the paper does not isolate or explain it. This creates an intriguing open question: does progressive context scaling teach the model generalizable grounding strategies that work at any length (attention over relevant spans, efficient information routing), or does it simply expand the model's comfortable operating range to ~60K with residual pretraining knowledge handling the longer tail? The distinction matters because the answer determines how far training length can lag behind deployment length—if the mechanism is strategy learning, the gap could be large; if it's range extension, the gap is bounded by the model's pretraining context exposure. This question, unresolved in the current paper, will shape how future work designs context-length curricula.

The role of verifier quality is reframed—but not resolved—for long-context settings. The paper's hybrid reward function (Equation 14) uses a small judge model (Qwen2.5-1.5B) during training and a large judge (DeepSeek-V3) during evaluation, creating an implicit assumption that the small judge's assessments are sufficiently correlated with the large judge's to provide a useful training signal. The paper's strong results suggest this assumption holds for DocQA, but the absence of judge agreement analysis means we cannot characterize when it holds or why. This opens a research direction parallel to verifier over-optimization in short-context settings: in long-context reasoning RL, the verifier must evaluate answers whose correctness depends on information distributed across a long context, which may require different judge capabilities than short-context answer verification. A small judge that works for DocQA may fail for long-context tasks requiring deeper semantic understanding (e.g., legal reasoning where answers are paragraphs rather than short spans).

Research directions that become more attractive: (1) principled SFT-RL calibration—determining the optimal SFT checkpoint for any given model-task pair without running full RL pipelines, perhaps through entropy-based early stopping criteria; (2) input-length-aware RL algorithms that automatically adjust clipping, learning rates, or advantage normalization based on the input length distribution rather than requiring a manual curriculum; (3) length generalization diagnostics—stratified evaluation that reveals whether progressive scaling teaches strategies or merely extends range; (4) training-dynamics-aware verifier design for long-context tasks—judge models whose reliability is explicitly characterized as a function of context length and answer complexity.

Research directions that become less attractive: (1) Blindly applying short-context RL algorithms to long-context tasks without addressing the identified instability mechanisms—Figure 2 provides a clear diagnostic that such approaches will likely fail; (2) treating SFT as a maximization problem where more SFT data and longer SFT training are always better—Figure 6 shows this can actively harm downstream RL; (3) developing increasingly complex RL algorithms for long-context reasoning without first addressing the fundamental stability issues that progressive scaling targets—the paper's GRPO vs. DAPO comparison shows the framework matters more than algorithmic nuances.

Follow-Up Research This Work Enables

Stratified length generalization analysis to determine the mechanism of progressive scaling. The paper demonstrates that QWENLONG-L1 performs well on benchmarks with maximum context lengths up to 176K tokens despite training only up to 60K tokens, but it never reports performance broken out by input length bucket. A direct follow-up would evaluate QWENLONG-L1-32B on the DocMath and Frames benchmarks (the two with examples exceeding 60K) and plot accuracy as a function of input length, with separate curves for the base model, the SFT model, and the final RL model. If accuracy remains flat or declines gradually beyond 60K, the mechanism is likely strategy generalization—the model learned grounding approaches that work at any length. If accuracy drops sharply at a threshold near 60K, the mechanism is likely range extension—the model's comfortable operating window was expanded but not fundamentally transformed. The same experiment should be run with an ablation where Phase II uses 120K instead of 60K to test whether training closer to the evaluation maximum changes the generalization curve. This experiment would directly inform how future progressive scaling curricula should set their phase thresholds relative to target deployment lengths.

SFT-RL calibration via entropy-based early stopping. The Figure 6 result—that long-context SFT reduces RL gains by ~10×—raises a practical question: can we predict the optimal SFT checkpoint without running RL? A concrete experiment would train SFT for a range of checkpoints (1, 2, 3, 5, 10 epochs on the 5.3K SFT dataset) and for each checkpoint, measure (a) validation accuracy on the DocQA benchmarks, (b) output entropy on a held-out set of long-context inputs (measured as the average token-level entropy of the model's output distribution), and (c) the KL divergence from the base model. Then run RL from each checkpoint (using the paper's Phase I → Phase II protocol) and measure the RL gain. The hypothesis is that RL gain is predictable from pre-RL metrics—specifically, that higher pre-RL entropy and higher KL divergence from the base model (indicating the SFT has not overly narrowed the distribution) predict larger RL gains, while validation accuracy alone does not. This would provide a practical diagnostic: before committing to expensive RL training, check whether the SFT model's output distribution still has sufficient entropy. If validated, this diagnostic would become a standard part of the reasoning model training pipeline, analogous to how pretraining scaling laws use validation loss to predict downstream performance.

Cross-model-family replication to test generality of the progressive scaling framework. The paper's entire experimental corpus uses R1-Distill-Qwen models. A critical stress-test would replicate the QWENLONG-L1 training recipe (warm-up SFT → Phase I 20K RL → Phase II 60K RL with retrospective sampling) using a different base model family—for example, LLaMA-3-based reasoning models (if available) or DeepSeek-R1-Distill-Llama variants. The key measurements are: (a) does the Figure 2 diagnostic (entropy collapse, KL instability on long-context RL) replicate in the new model family? (b) does progressive scaling provide similar stabilization benefits? (c) does the SFT-RL tradeoff (Figure 6) replicate, or is it specific to Qwen's pretraining? A negative result—progressive scaling not helping or the SFT-RL tradeoff reversing—would be equally valuable, as it would establish boundary conditions on the framework's applicability and suggest that Qwen's specific pretraining (architecture, data mixture, position encoding) interacts with the training dynamics. A positive result across multiple model families would substantially strengthen the paper's claim of providing a "fundamental recipe."

Comparing progressive context scaling against alternative stabilization methods. The paper demonstrates that phased RL stabilizes training, but it does not test whether other stabilization techniques could achieve similar effects without a curriculum. A controlled experiment would compare four approaches to long-context RL, all starting from the same SFT model and using the same total compute: (1) the paper's Phase I → Phase II curriculum (20K → 60K), (2) single-stage RL with a lower learning rate (e.g., 1e-6 instead of 2e-6) to reduce update variance, (3) single-stage RL with stronger KL regularization (adding back the KL penalty that the paper removed from GRPO), and (4) single-stage RL with gradient accumulation over larger batches to reduce gradient variance. The outcome metrics would be final accuracy, entropy trajectory, and KL divergence stability. This experiment would distinguish whether progressive scaling is necessary (because input-length heterogeneity creates pathologies that no amount of hyperparameter tuning can fix) or merely sufficient (it's one way to stabilize training, but others work too). If alternative stabilization methods match the curriculum's performance, the practical recommendation shifts from "use progressive scaling" to "use whatever stabilization method fits your infrastructure."

SFT-RL tradeoff analysis for the 32B model scale. The paper's Figure 6 experiment, demonstrating the SFT-RL tradeoff where long-context SFT reduces subsequent RL gains, was conducted only on the 14B model. A critical open question is whether this tradeoff scales with model size—do larger models, which may have stronger inherent reasoning capabilities and different SFT dynamics, show the same entrapment in local optima? A direct replication of Figure 6 at 32B scale (base model → RL, short-context SFT → RL, long-context SFT → RL) would reveal whether the optimal SFT strategy is model-size-dependent. Larger models might benefit more from SFT (because they have more capacity to absorb supervised patterns without over-constraining exploration) or might show the same tradeoff (because the mechanism—SFT narrowing the output distribution—is scale-invariant). The 32B SFT model already achieves 68.7 (Table 4), only 2.0 points below the RL-augmented 70.7, so the RL headroom is smaller. Understanding whether this smaller headroom is a scaling law effect (larger models extract more from SFT, leaving less for RL) or a ceiling effect (the benchmarks have limited room for improvement) would inform whether future work should invest in larger models with more SFT or in better RL methods for extracting gains from already-strong SFT models.

Dynamic difficulty estimation using online reward tracking. The paper's difficulty-aware retrospective sampling uses fixed difficulty scores computed from the base model before training. A natural extension would replace this with online difficulty tracking: during Phase I RL, continuously update each example's difficulty score based on the current policy's rolling average reward, and use these updated scores to select examples for retention in Phase II. The hypothesis is that online tracking would better identify examples that remain at the policy's learning frontier—examples that were hard for the base model but became easy during Phase I would be dropped, while examples that remain hard despite Phase I training would be retained. The experiment would compare fixed difficulty (base model scores, as in the paper) against online difficulty (updated every N steps during Phase I) in terms of (a) which examples are retained, (b) Phase II entropy and reward trajectories, and (c) final benchmark performance. A positive result (online tracking improves final performance) would suggest that the current fixed-difficulty approach is leaving performance on the table by retaining examples the policy has already mastered.

Practical Applications and Downstream Use Cases

Financial document analysis at scale. The paper's strongest results come on DocMath (+5.0 to +5.5 points over base for the 32B model, reaching 67.5–68.0 and surpassing Claude-3.7-Sonnet-Thinking at 67.5) and involve exactly the kind of reasoning that financial analysts perform: extracting numerical values from dense SEC filings, bond prospectuses, and financial reports, then computing derived quantities like total capital costs or accrued interest (as illustrated in the case studies). A concrete deployment would use QWENLONG-L1-32B to automate the initial analysis of quarterly earnings reports, merger agreements, or debt offering documents—tasks that currently require human analysts to read hundreds of pages of legalese and financial tables to extract and compute key metrics. The model's demonstrated ability to ground specific numerical values (e.g., "$300.0 million in aggregate principal amount of 8.0% senior unsecured notes") while ignoring distracting details (as shown in case study complong-testmini-183, where QWENLONG-L1-14B correctly filters out irrelevant temporal information that misleads the base model) directly addresses the most common failure mode in financial NLP: extracting the right numbers from the wrong context. The 120K evaluation context length comfortably covers most financial documents (a typical 10-K filing is 50K–100K tokens), and the Pass@K scaling (reaching ~80% at Pass@16) means that running multiple samples with a simple answer-selection mechanism could push accuracy high enough for production use in analyst-assistance workflows rather than fully autonomous decision-making.

Multi-hop reasoning over legal and regulatory corpora. QWENLONG-L1-32B achieves 90.5% on 2WikiMultihopQA—near-ceiling performance on a task that requires synthesizing information across multiple documents to answer questions that no single document contains. This capability transfers directly to legal research, where practitioners must trace precedents across case law, connect statutory language to regulatory interpretations, and verify whether a specific fact pattern satisfies multi-factor legal tests. A deployment would integrate QWENLONG-L1 into a legal research pipeline that: (1) retrieves potentially relevant cases and statutes for a given legal question, (2) presents the full text of these documents as context (with typical legal documents ranging from 5K–50K tokens each, and multi-document queries easily exceeding 60K tokens total), and (3) asks the model to reason about how the retrieved authorities apply to the specific facts. The model's demonstrated grounding and backtracking behaviors (case study complong-testmini-265 showing the model correctly computing interest across a two-year period while verifying against multiple document sections) suggest it would handle the common legal reasoning pattern of "this case says X, but the facts here differ in way Y, so X may not apply, requiring analysis of alternative case Z." The key practical benefit is reducing the time attorneys spend on initial case law synthesis—currently hours of manual reading and cross-referencing—to a model-assisted process where the human verifies the model's reasoning rather than constructing it from scratch.

Scientific literature review and evidence synthesis. The strong performance on Qasper (58.5%, +8.5 over base, the highest among all models including proprietary ones) is particularly relevant because Qasper tests question-answering over NLP research papers—exactly the kind of dense, technical, multi-section documents that define scientific literature. A deployment would use QWENLONG-L1 to answer specific scientific questions by reasoning across multiple papers: e.g., "What is the reported improvement in BLEU score for method X compared to baseline Y, and what datasets were these results measured on?"—a question requiring the model to ground specific numbers from specific sections of specific papers, compare them, and produce a synthesized answer. The model's ability to handle context lengths up to 120K tokens means it can process 3–5 full research papers simultaneously (typical NLP papers are 8K–15K tokens), enabling cross-paper synthesis that goes beyond single-paper QA. The practical benefit is accelerating systematic reviews and meta-analyses where researchers currently manually extract and compare results across dozens of papers—a process that takes weeks to months. QWENLONG-L1 could serve as a first-pass extraction and synthesis engine, with human researchers verifying and correcting its outputs rather than starting from scratch.

Test-time compute scaling for high-stakes single-query scenarios. The paper's Pass@K analysis (Figure 4) shows that sampling 16 outputs and selecting the best one improves accuracy from ~68–69 (Pass@1) to ~80 (Pass@16) for QWENLONG-L1-14B. For applications where a single query's correctness is highly consequential—medical guideline interpretation, compliance verification, contract risk assessment—this test-time scaling provides a practical deployment strategy: run the model 16 times, use a strong judge (DeepSeek-V3 or equivalent) to score each answer, and take the highest-scoring response or the majority-consensus answer. The computational cost is 16× the single-query cost, but for low-volume, high-stakes queries, this is often acceptable. Importantly, the paper's finding that Pass@K continues to improve without plateauing at K=16 suggests further gains are available at higher sample counts, and the model's training specifically encourages diverse exploration (through temperature 0.7 sampling and the removal of the KL penalty during RL), meaning the 16 outputs are likely to explore genuinely different reasoning paths rather than producing minor variations on the same answer. This deployment pattern is directly applicable to the deep research use cases the paper cites as motivation—when an AI system is autonomously collecting and synthesizing information to produce a research report, each factual claim or numerical computation within that report could be verified by sampling multiple reasoning chains and checking for consistency.