The Unlearnability Phenomenon in RLVR for Language Models

🎯 Pitch

Even when language models stumble upon correct answers during reasoning training, a large fraction of hard problems remain stubbornly unlearnable. The root cause is not optimization but a fundamental representation flaw—these examples are gradient outliers with incoherent internal reasoning that RL post-training alone cannot fix.

1. Executive Summary

This paper studies a counterintuitive failure mode in Reinforcement Learning with Verifiable Reward (RLVR) training for LLMs, revealing that a substantial subset of initially difficult math problems on MATH and DeepScaleR datasets remain unlearnable even when correct rollouts are consistently sampled during GRPO training on Qwen and Llama models. The authors systematically rule out common optimization-side explanations—positive rollout scarcity (oversampling with replay), gradient regularization (clipping and KL penalty ablation), and gradient interference (within-prompt and cross-prompt cosine similarity analysis)—before demonstrating through cross-example gradient analysis that unlearnable examples are gradient outliers with fundamentally flawed internal representations, characterized by low gradient similarity to the broader training distribution and ungeneralizable reasoning patterns (coherent final answers paired with incoherent intermediate steps). Data augmentation and curriculum learning fail to improve gradient similarity or reasoning quality during the RL stage, while mid-training on the base model substantially increases gradient alignment, establishing that representation flaws causing unlearnability are largely irreparable through RL post-training alone.

2. Context and Motivation

The Core Problem: Positive Reward Doesn't Guarantee Learning

The dominant paradigm for improving LLM reasoning over the past two years has been Reinforcement Learning with Verifiable Reward (RLVR) — train a model to maximize binary rewards based on whether its final answer matches a ground-truth answer, using algorithms like GRPO (Shao et al., 2024). The underlying assumption is straightforward and appealing: if the model can occasionally stumble into a correct solution (receiving a positive reward signal), and can also see its own incorrect attempts (receiving negative signals), then the contrast between these outcomes should reinforce whatever reasoning patterns led to success and suppress those that led to failure. Under this assumption, any example where correct rollouts exist during training should eventually be learned.

This paper confronts that assumption directly and finds it to be false. As demonstrated in Figure 1b, a substantial fraction of training examples — those the model initially struggles with — exhibit a training reward curve that stays stubbornly flat, never rising above baseline levels even after convergence when validation performance saturates. These examples are not starved of positive reward: the authors explicitly filter out examples that never observe any correct rollouts during training (Section 3.2), meaning the unlearnable subset does receive positive signals. Yet the model fails to internalize them. The paper terms this the unlearnability phenomenon, and its central question is deceptively simple: why do certain examples remain unlearnable despite receiving positive reward signals?

This gap matters because it exposes a fundamental limitation in how outcome-based RL transfers learning. If the mechanism that RLVR relies on — the positive-negative contrast — doesn't actually work for a large fraction of the training distribution, then the field's understanding of what RLVR does and how to improve it is incomplete in critical ways.

Why This Problem Is Important

The importance of understanding unlearnability spans both practical deployment considerations and theoretical questions about what RLVR actually learns.

Practical significance: wasted training resources on dead weight data. If 16–30% of difficult training examples are fundamentally unlearnable under GRPO (as Table 1 shows — 16.7% for Qwen2.5-3B on DeepScaleR, 30.2% for Qwen2.5-0.5B on MATH Easy, 21.9% for Llama-3.2-3B-Instruct on MATH Hard), then a non-trivial fraction of training compute is being spent sampling, scoring, and computing gradients for examples that will never contribute to downstream performance. Worse, these examples may be actively harmful: they consume training cycles that could have been spent on learnable examples, they may introduce noisy gradients that interfere with learning on other data, and they can create false confidence that continued training will eventually yield improvements when in fact it won't. Identifying which examples are unlearnable — and understanding why — could enable smarter data selection, curriculum design, or even targeted pretraining interventions that make these examples learnable before the RL stage begins.

Theoretical significance: the relationship between reward and learnability is poorly characterized. GRPO (and PPO-style algorithms more broadly) are designed under the assumption that positive advantage gradients steer the policy toward better outputs. But if the mere presence of correct rollouts is insufficient for learning on some examples, then something beyond reward availability constrains learning — and that "something" is currently opaque. The paper's gradient analysis (Section 5.1) points toward the answer: learnability is correlated with gradient similarity to the broader training distribution. In other words, RLVR doesn't just require positive reward — it requires that the gradient signal from that reward aligns with what the model learns from other examples. When an example is a gradient outlier, its positive signal is isolated and fails to reinforce or be reinforced by learning on other data. This transforms the question from "does this example have correct rollouts?" to "does this example's gradient landscape support transfer from the rest of the training distribution?" — a much deeper and more theoretically interesting framing.

Broader implications for the RLHF and reasoning pipeline. RLVR is not just one technique among many — it is the core method behind DeepSeek-R1 (Guo et al., 2025), Open-R1 (Hugging Face, 2025), and most state-of-the-art reasoning models. If the unlearnability phenomenon is widespread (and the paper shows it across three model families and two datasets), it suggests that current RLVR pipelines may be fundamentally limited in what subset of the training distribution they can improve on, even with unlimited sampling budgets. The finding that mid-training can reshape representations to increase gradient similarity (Section 5.4, Figure 11) points toward a revised training pipeline where the role of RL is not to fix flawed representations but to polish already-aligned ones — with mid-training or pretraining doing the heavy lifting of ensuring examples are learnable before RL begins. This would represent a significant shift in how the field allocates compute across training stages.

Prior Approaches and Where They Fall Short

The prevailing view: reward availability determines learnability. A substantial body of recent work has focused on the question of whether and how LLMs learn new skills during RLVR (Yue et al., 2025; Liu et al., 2025a; Wu et al., 2026). The initial provocations came from findings that pass@k (the probability that at least one of k independent samples is correct) degrades after RL training (Yue et al., 2025), suggesting that models aren't truly acquiring new capabilities but rather sharpening an existing distribution. Follow-ups explored what exactly models learn: Yuan et al. (2025) and Park et al. (2025) found evidence that RL teaches models to compose atomic skills acquired during supervised fine-tuning rather than discovering entirely new solution strategies. Wu et al. (2026) provided theoretical arguments and empirical evidence for why RLVR cannot help models escape their initial policy distribution to find genuinely novel solutions.

These works converge on a shared implicit model of RLVR's learning dynamics: if the base model has some probability of producing a correct answer on an example (i.e., pass@k > 0), then RL can amplify that signal and increase the probability of correctness. The limiting factor, in this view, is exploration — whether correct rollouts are sampled frequently enough during training to provide sufficient gradient signal. This has motivated a line of work on improving exploration and credit assignment: oversampling difficult examples (Yu et al., 2025; Sun et al., 2025b; Zhang et al., 2025d;c), entropy-based loss weight adjustment (Cui et al., 2025; Cheng et al., 2025), and fine-grained credit assignment through token-level or step-level advantage (Liu et al., 2025b; Zheng et al., 2025a).

Where this view falls short: it cannot explain unlearnability. The paper's findings on unlearnability represent a direct challenge to the exploration-as-bottleneck model. In Section 4.1, the authors address the most natural hypothesis — that unlearnable examples simply don't see enough positive rollouts — by implementing oversampling with experience replay that guarantees exactly $k_{pos}=1$ correct rollout per batch per example. The result (Figure 2): it slows down learning on the learnable group (because constraining positive examples artificially reduces the reward variance that GRPO exploits) but does nothing to help the unlearnable group. The gap persists. Even stronger interventions — supervised fine-tuning on distilled correct responses and RL with an enormous rollout group ( $k=64$ ) on unlearnable examples alone (Appendix A.2, Figure 15) — fail to close the gap. This is direct evidence that something more fundamental than reward availability is at play.

Optimization-side fixes also fail. The paper tests whether regularization mechanisms in standard PPO/GRPO — specifically, the clipping term that suppresses gradients for low-probability tokens and the KL penalty that constrains deviation from the reference model — are responsible (Section 4.2). If unlearnable examples produced correct rollouts that were systematically less probable under the reference policy, their gradients would be disproportionately suppressed. But Figure 3 shows reference log-likelihood distributions are comparable across all groups, and Figure 4 shows clipping rates track closely. Ablating clipping (clip-higher) and removing the KL penalty entirely (Figure 5) produces no improvement. Gradient interference — where gradients from correct and incorrect rollouts cancel each other because they share critical tokens (Nguyen et al., 2025) — is also ruled out (Appendix A.3, Figure 16): gradients on unlearnable examples remain aligned between correct and incorrect rollouts rather than opposing each other, meaning interference cannot explain the failure to learn.

The closest prior work: Sun et al. (2025a) on extremely difficult examples. The most directly related work is Sun et al. (2025a), who study the learning dynamics of "extremely difficult" examples with near-zero initial pass@k. They find that such examples can be learned if fine-grained reward assignment (beyond binary outcome reward) is available — for example, rewarding partial progress or intermediate steps. This is an important result that seems superficially at odds with the unlearnability phenomenon. However, the two findings are complementary rather than contradictory: Sun et al. show that with sufficiently rich reward signals, even very hard examples become learnable, while this paper shows that with only binary outcome reward, some examples remain stubbornly unlearnable even when correct rollouts exist. The unlearnability phenomenon thus identifies a boundary condition on Sun et al.'s positive result — outcome reward alone is insufficient for a non-trivial fraction of examples, and the reason traces back to representation quality rather than reward sparsity.

Data scheduling and curriculum learning don't address the root cause. Recent work has explored dynamic sampling (Yu et al., 2025) and curriculum learning (Shi et al., 2025; Gao et al., 2025) to improve RLVR efficiency by training on easier examples first and progressively introducing harder ones. The paper's curriculum learning experiment (Appendix A.5, Figure 18) tests whether this approach can bootstrap learning on unlearnable examples. It cannot: even after the model has substantially improved on easy and learnable data, when unlearnable examples are introduced, their reward stays flat and validation performance drops. The skills acquired on learnable data simply don't transfer.

How This Paper Positions Itself

The paper frames itself as a systematic characterization of an overlooked failure mode rather than a proposal for a new method. Its contributions are diagnostic and analytical:

Phenomenon identification. The paper is the first to explicitly define and quantify the unlearnability phenomenon in RLVR, establishing that it exists across models, datasets, and training configurations (Section 3.3, Table 1). This is a necessary first step — before the field can address unlearnability, it must recognize that it's a real and consistent effect, not an artifact of a particular training run or hyperparameter choice.
Hypothesis elimination. The paper systematically rules out the most natural candidate explanations — reward scarcity, gradient regularization, and gradient interference (Sections 4.1, 4.2, Appendix A.3) — using targeted interventions. This narrows the search space for the true cause and prevents future work from wasting effort on dead-end hypotheses.
Root cause identification through gradient analysis. The paper's central analytical contribution is the cross-example gradient similarity analysis (Section 5.1), which reveals that unlearnable examples are gradient outliers — their optimization signals are poorly aligned with the rest of the training distribution. This provides a mechanistic explanation for unlearnability: the model cannot transfer what it learns from other examples to these outliers, and the isolated positive signals on these examples are too weak to drive learning on their own.
Demonstration that representation flaws are RL-irreparable. The paper shows that data augmentation (generating similar problems and subproblems) and curriculum learning fail to improve gradient similarity or reasoning quality on unlearnable examples (Sections 5.3, Appendix A.5), while mid-training can increase gradient alignment (Section 5.4, Figure 11). This establishes a clear division of labor: representation flaws that cause unlearnability are difficult or impossible to fix during the RL post-training stage, but they are addressable through pretraining-stage interventions. This has direct implications for how training pipelines should be designed.
Connection to reasoning quality. The paper goes beyond gradient analysis to examine the actual reasoning traces (Section 5.2), finding that unlearnable examples often have correct final answers paired with incoherent or erroneous intermediate steps (Table 2). This connects gradient properties to observable behavior — the model is producing "correct" answers through ungeneralizable shortcuts or heuristics (what Nikankin et al., 2025, call a "bag of heuristics"), and the binary outcome reward cannot distinguish genuine reasoning from lucky guessing.

The paper explicitly does not claim to provide a solution to unlearnability. It instead argues that the phenomenon represents a fundamental limitation in current RL approaches for reasoning tasks, and that addressing it will require investing more heavily in the stages of training before RL — pretraining and mid-training — where representations can be shaped to ensure that difficult examples are learnable when RL begins. The findings thus reframe the conversation around RLVR from "how do we improve the RL algorithm?" to "what makes an example learnable under RL in the first place, and how do we ensure our training data meets those criteria?"

3. Technical Approach

3.1 Reader Orientation

This is a diagnostic investigation that builds a measurement and analysis pipeline to characterize why certain training examples in LLM reinforcement learning fail to be learned. The "system" is not a new algorithm but rather a suite of analytical tools — gradient similarity computation, reasoning quality annotation, and controlled training interventions — that together isolate the root cause of the unlearnability phenomenon as a representation-level issue rather than an optimization-side problem.

3.2 Big-Picture Architecture (Diagram in Words)

The paper constructs a five-part analytical framework:

Example Categorization Pipeline — Classifies training examples into easy, learnable, and unlearnable groups based on initial success rate and training reward dynamics under GRPO.
Hypothesis Testing Suite — Systematically evaluates four candidate explanations (positive rollout scarcity, gradient regularization, gradient interference, representation flaws) through targeted training interventions and gradient measurements.
Gradient Similarity Analyzer — Computes example-level gradients from correct rollouts and measures their cosine similarity against the broader training distribution, using a fixed LoRA adapter for computational efficiency.
Reasoning Quality Evaluator — Uses GPT-5-mini to score intermediate reasoning traces from correct-answer rollouts on a 0–5 scale, connecting gradient properties to observable behavior.
Intervention Testing Harness — Applies data augmentation (similar problem generation, subproblem decomposition) and curriculum learning to test whether representation flaws can be repaired during the RL stage.

Information flows: training data → example categorization via pass@1 thresholds and reward trajectory analysis → gradient computation on correct rollouts → similarity measurement across examples → reasoning quality annotation → controlled intervention experiments to test causality.

3.3 Roadmap for the Deep Dive

First, the working definition of "unlearnable" examples (Section 3.2), which operationalizes a fuzzy concept into a concrete threshold-based criterion that can be applied consistently across models and datasets. This is the foundation everything else builds on.
Second, the GRPO training algorithm and dynamic sampling (Section 3.1), since all experiments use this as the baseline and the interventions in Section 4 modify specific components of this algorithm.
Third, the oversampling-with-replay mechanism (Section 4.1, Algorithm 1), which is the paper's primary tool for testing the positive-rollout-scarcity hypothesis and illustrates the methodology of controlled intervention.
Fourth, the gradient similarity computation pipeline (Section 5.1), including the LoRA-based approximation and the cosine similarity metric, since this is the paper's central analytical technique and the evidence it produces drives the main conclusions.
Fifth, the reasoning quality evaluation protocol (Section 5.2), which connects gradient outliers to observable reasoning flaws.
Sixth, the data augmentation procedures (Section 5.3, Appendix C), which test whether representation issues can be fixed by adding semantically similar training data.
Seventh, the mid-training analysis (Section 5.4), which provides the positive contrast — showing that representation flaws are fixable, but only through pretraining-stage interventions.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical diagnostic paper whose core idea is that certain training examples resist learning under RLVR despite receiving positive reward signals, and that this resistance traces to flawed internal representations — gradient outliers — that cannot be repaired during the RL post-training stage.

GRPO Baseline and Dynamic Sampling

Algorithm. The paper uses Group Relative Policy Optimization (GRPO) with dynamic sampling as its baseline RL algorithm (Section 3.1). GRPO is a variant of PPO designed specifically for LLM fine-tuning with verifiable rewards. Unlike standard PPO, which requires a learned value function, GRPO estimates advantages using only the reward variance within a group of rollouts for the same prompt. This removes the need for a separate critic model, simplifying the training pipeline.

For each training example $(x, y^*)$ consisting of a prompt $x$ and ground-truth answer $y^*$ , the model samples $k$ responses $\{y_i\}_{i=1}^k$ from the current policy $\pi_{\theta_{\text{old}}}$ . Each response is automatically verified against the ground truth, producing a binary reward $1[y_i = y^*]$ — 1 if the final answer matches, 0 otherwise.

Advantage computation. The advantage for each rollout is computed as the standardized reward within the group:

$\hat{A}_i = \frac{1[y_i = y^*] - \text{mean}(\{1[y_i = y^*]\}_{i=1}^k)}{\text{std}(\{1[y_i = y^*]\}_{i=1}^k)}$

where $\text{mean}(\{1[y_i = y^*]\}_{i=1}^k)$ is the empirical mean reward across the $k$ rollouts for this prompt, $\text{std}(\{1[y_i = y^*]\}_{i=1}^k)$ is the empirical standard deviation of those rewards, and $\hat{A}_i$ is the resulting scalar advantage for rollout $i$ .

What this computes: For each rollout in the group, it measures how much better or worse that rollout's reward is compared to the average reward for this prompt, expressed in units of standard deviation. Rollouts that are correct when most are incorrect receive large positive advantages; correct rollouts when most are correct receive small positive advantages; incorrect rollouts when most are correct receive negative advantages. This makes the gradient update proportional to how surprising the outcome is relative to the current policy's performance on that specific prompt — a form of local normalization that adapts to varying difficulty.

Why this form: Standardizing by the in-group mean and standard deviation removes the need for a global value function baseline (which would be hard to learn and would add computational cost) while still providing the variance reduction that makes policy gradient methods stable. The group-based normalization means that on easy prompts (where most rollouts are correct), even a correct rollout produces a small advantage, preventing the model from over-optimizing on already-solved problems. On hard prompts (where most rollouts are wrong), a rare correct rollout produces a large advantage, strongly reinforcing whatever led to the success.

Policy optimization. The policy is optimized to maximize the clipped PPO objective:

$\mathcal{L}_{\text{GRPO}}(\theta, (x, y^*)) = -\frac{1}{k} \sum_{i=1}^{k} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left(r_{i,t} \hat{A}_i, \text{clip}(r_{i,t}, 1 - \varepsilon, 1 + \varepsilon) \hat{A}_i\right) - \beta \text{KL}(\pi_\theta | \pi_{\text{ref}})$

where $k$ is the number of rollouts per prompt (default 8), $|y_i|$ is the length in tokens of rollout $i$ , $r_{i,t} = \frac{\pi_\theta(y_{i,t} | x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} | x, y_{i,<t})}$ is the probability ratio between the current and old policies for token $t$ of rollout $i$ , $\varepsilon$ is the clipping threshold (typically 0.2), $\beta$ is the KL penalty coefficient, and $\text{KL}(\pi_\theta | \pi_{\text{ref}})$ is the Kullback-Leibler divergence between the current policy and a frozen reference model (typically the initial policy).

What this computes: For each token in each rollout, it computes the product of the probability ratio $r_{i,t}$ and the rollout-level advantage $\hat{A}_i$ . If $r_{i,t} \hat{A}_i$ is within the clipping range $[1-\varepsilon, 1+\varepsilon] \cdot \hat{A}_i$ , the full product is used. If $r_{i,t}$ has moved too far from 1 (meaning the current policy assigns very different probability to this token than the old policy), the gradient contribution is clipped, preventing large destructive updates from individual tokens. The $\min$ operator ensures we always take the more conservative (closer-to-zero) of the clipped and unclipped values. The KL penalty term $\beta \text{KL}(\pi_\theta | \pi_{\text{ref}})$ is subtracted, penalizing the policy for deviating too far from the reference model. The outer $\frac{1}{k} \sum_{i=1}^k$ averages across rollouts, and the inner $\frac{1}{|y_i|} \sum_{t=1}^{|y_i|}$ averages across tokens within each rollout.

Why this form: The PPO clipping mechanism is the standard solution to the trust-region problem in policy optimization — it prevents the policy from changing too much in a single update by capping the effective gradient when probability ratios become extreme. This is particularly important for LLMs, where token probabilities can shift dramatically during training. The KL penalty provides an additional regularization against distributional collapse. The token-level averaging with rollout-level advantage is a deliberate simplification: it assigns the same advantage to every token in a rollout, treating the entire response as the unit of credit rather than attempting to identify which specific tokens were responsible for the outcome. This avoids the need for a process reward model while still providing a coherent learning signal — correct rollouts get positive advantage across all their tokens, incorrect ones get negative advantage.

Dynamic sampling. The paper uses dynamic sampling (DAPO-style filtering; Yu et al., 2025) to improve training efficiency. At each training step, prompts where all $k$ rollouts produce the same outcome (all correct or all incorrect, yielding zero reward variance and thus undefined standardized advantage) are filtered out of the gradient update:

$\mathcal{D}_{\text{update}} = \{(x, y^*) \mid \text{std}(\{1[y_i = y^*]\}_{i=1}^k) \neq 0\}$

where $\mathcal{D}_{\text{update}}$ is the set of prompts that actually contribute to the gradient at this step.

What this computes: A simple binary filter — only prompts with at least one correct and one incorrect rollout participate in gradient updates. Prompts that are already perfectly solved (all correct) or currently unsolvable (all incorrect) are skipped for that step.

Why this form: Prompts with zero reward variance contribute zero advantage to all rollouts (since the mean equals every individual reward, the numerator is zero for all rollouts), so they produce no useful gradient signal. Computing and discarding their gradients wastes compute. Dynamic sampling removes them before gradient computation, focusing the optimizer's effort on prompts where contrastive learning can actually occur.

Training hyperparameters. The paper uses several model-dataset configurations (Section 3.3, Appendix B.1):

Qwen2.5-0.5B on MATH Easy (levels 1–4): sampling batch size 1024, rollout group size $k=8$ , gradient update batch size 64 (effective rollout-level batch size 512), learning rate $5 \times 10^{-7}$ , max response length 5120 tokens.
Qwen2.5-3B on DeepScaleR (40k problems): sampling batch size 256, $k=8$ , gradient batch size 64, learning rate $1 \times 10^{-6}$ , max length 8192 tokens.
Llama-3.2-3B-Instruct on MATH Hard (levels 3–5): sampling batch size 256, $k=8$ , gradient batch size 64, learning rate $5 \times 10^{-7}$ , max length 5120 tokens.
All use temperature 1.0 for sampling, Math-Verify for answer verification, and verl for full-parameter RL fine-tuning with the reference model frozen at initialization.

Working Definition of Unlearnability

Operationalization. The paper defines unlearnable examples through a quantitative criterion applied after RL training converges (Section 3.2). An example is considered unlearnable if it satisfies two conditions:

Low final accuracy: The example's pass@1 under the final trained policy is below a threshold $\tau$ (specifically, $\text{pass@1} < 0.1$ ). Pass@1 is estimated by sampling $N=32$ responses from the final model and computing the fraction that produce the correct final answer.
Positive reward received during training: The example observed at least one correct rollout at some point during the GRPO training process. This ensures unlearnability is not simply caused by the model never accidentally producing a correct answer — these examples did have opportunities to learn from positive signals.

Additionally, the paper separates "hard" examples (initial success rate < 10%, estimated from the initial policy by sampling responses) from "easy" examples (initial success rate ≥ 10%). Unlearnable and learnable examples are both subsets of the hard group.

What this defines operationally: A three-way categorization of training data into:

Easy examples ( $\mathcal{D}_{\text{easy}}$ ): initial pass@1 ≥ 0.1. These saturate to high reward early in training.
Learnable examples ( $\mathcal{D}_l$ ): initial pass@1 < 0.1 but final pass@1 ≥ 0.1. These start hard but are successfully learned.
Unlearnable examples ( $\mathcal{D}_u$ ): initial pass@1 < 0.1 and final pass@1 < 0.1, despite observing correct rollouts during training.

Why this threshold-based definition: A sharp categorical distinction ("this example is unlearnable") is inherently an approximation — learnability is a continuous property, and examples near the threshold will have intermediate behavior. The paper acknowledges this limitation but argues that the operational definition provides a useful analytical tool for studying the tendency toward unlearnability at the extremes. The choice of $\tau=0.1$ is "rather arbitrary" (the paper's own words), but the key results are robust because (1) the analyses compare distributions of properties across groups, not just single examples, and (2) the paper uses intersection across three independent training runs to reduce noise — an example is only classified as unlearnable if it falls below the threshold in all three runs, and similarly for learnable. This intersection procedure removes examples whose classification is sensitive to training stochasticity.

Practical implementation. For each model-setting combination, the authors run GRPO training three times independently. After convergence, they sample 32 responses per example from the final policy, compute pass@1, and take the intersection across runs for $\mathcal{D}_u$ and $\mathcal{D}_l$ . Examples that never observe positive reward (union across three runs) are excluded from both groups. The resulting percentages (Table 1) represent a conservative estimate — the intersection criterion ensures that only consistently unlearnable examples are analyzed.

Exclusion of zero-positive-reward examples. The paper explicitly excludes examples that never observe any correct rollout throughout RLVR training (the "w/o Pos. Reward" column in Table 1). These examples are not considered unlearnable in the paper's framework because the failure to learn could be trivially explained by the absence of any positive signal — the model never saw what a correct answer looks like. The unlearnability phenomenon specifically refers to examples that do receive positive signals yet fail to benefit from them, which is the more puzzling and informative case.

Oversampling with Experience Replay (Hypothesis 1 Intervention)

Motivation. The most natural explanation for unlearnability is that unlearnable examples simply don't produce enough correct rollouts during training for the positive signal to accumulate meaningfully (Hypothesis 1, Section 4.1). Even though these examples sometimes yield correct answers, the imbalance between correct and incorrect rollouts might mean the positive gradient is overwhelmed by the negative signal from many more incorrect rollouts.

Intervention design. The paper implements oversampling with experience replay (Algorithm 1, Appendix B.2) to guarantee a fixed ratio of positive to negative rollouts per example in every batch. The procedure works as follows:

For each training example, sample $4k$ rollouts from the current policy (where $k=8$ is the standard rollout group size, so 32 rollouts are sampled per example). Separate these into positive rollouts (correct final answer) and negative rollouts (incorrect). If the example has at least $k_{\text{pos}}$ positive rollouts (the paper uses $k_{\text{pos}}=1$ ), randomly sample $k_{\text{pos}}$ from them and $k - k_{\text{pos}}$ from the negatives. If the example has fewer than $k_{\text{pos}}$ positive rollouts, use all available positives and fill the remaining slots by replaying previously sampled positive rollouts from a buffer, with each buffered rollout replayed at most twice. The final batch for gradient computation contains exactly $k_{\text{pos}}$ positive and $k - k_{\text{pos}}$ negative rollouts per prompt. The advantage is then computed after this oversampling and replay, so the advantage calculation sees the controlled ratio.

What this guarantees: Every prompt contributes exactly one correct rollout and seven incorrect rollouts to each gradient update, regardless of how rare correct rollouts actually are for that prompt under the current policy. This eliminates reward scarcity as a confound — if unlearnable examples truly fail only because they see too few positive rollouts, this intervention should close the gap.

Why $k_{\text{pos}}=1$ : The paper focuses on this setting because it represents a realistic regime where the replay rate is already high for unlearnable examples (meaning the buffer is frequently needed to meet the one-positive-rollout quota). Larger $k_{\text{pos}}$ would require even more replay and larger $4k$ sampling cost — the setting chosen balances experimental control against computational feasibility.

What the result means: Figure 2 shows that controlling the positive-to-negative ratio slows down learning on the learnable group (because artificially constraining positives reduces the naturally higher reward variance these examples develop as they improve), but has no effect on the unlearnable group — their reward curve remains flat. This is strong evidence against Hypothesis 1 and motivates the search for deeper causes.

Regularization Ablation Experiments (Hypothesis 2 Interventions)

Motivation. Two mechanisms in standard GRPO could disproportionately suppress learning on unlearnable examples (Hypothesis 2, Section 4.2): (1) the PPO clipping term $\text{clip}(r_{i,t}, 1-\varepsilon, 1+\varepsilon)$ caps gradient contributions when the probability ratio $r_{i,t}$ is extreme, and (2) the KL penalty $-\beta \text{KL}(\pi_\theta|\pi_{\text{ref}})$ penalizes any deviation from the reference model. If correct rollouts on unlearnable examples were systematically less probable under the reference policy (i.e., the model's initial guess was far from the correct answer), then the gradient signal from those rollouts would be heavily clipped or penalized, preventing learning.

Diagnostic measurements. The paper conducts two checks before running full interventions:

Reference log-likelihood of correct rollouts (Figure 3): For each example group, the paper samples correct rollouts from the initial policy and computes $\log \pi_{\text{ref}}(y_{\text{correct}} | x)$ — the log-probability the reference model assigns to the correct answer. The distributions across unlearnable, learnable, and easy groups are plotted as violin plots.
Clipping rates during training (Figure 4): During GRPO training, the paper tracks what fraction of tokens in each group's rollouts have their probability ratio $r_{i,t}$ fall outside the clipping range $[1-\varepsilon, 1+\varepsilon]$ , causing their gradient contribution to be clipped. This is plotted as a time series across training steps.

What these measure: The first tells us whether the reference model starts with systematically lower confidence on unlearnable examples (which would make them more susceptible to clipping and KL penalty). The second tells us whether clipping actually occurs more frequently on unlearnable examples during training.

Results. Figure 3 shows the distributions are comparable — unlearnable examples do not have notably lower reference probabilities than learnable or easy examples. Figure 4 shows clipping rates track closely across all three groups, with all groups hovering around 0.1–0.5% throughout training. Both checks suggest Hypothesis 2 is unlikely.

Interventions. To confirm, the paper runs two ablations of the GRPO algorithm (Figure 5):

Clip-higher: Following DAPO (Yu et al., 2025), the clipping upper bound is raised from $1+\varepsilon$ to a higher value, effectively disabling the clipping penalty for positive advantages (since $r_{i,t} \hat{A}_i > (1+\varepsilon)\hat{A}_i$ can now still contribute gradient when $\hat{A}_i > 0$ ). This allows the model to aggressively increase the probability of tokens from correct rollouts without being capped.
No KL loss: The KL penalty term $-\beta \text{KL}(\pi_\theta|\pi_{\text{ref}})$ is removed entirely, allowing the policy to deviate arbitrarily from the reference model.

What these test: If clipping or KL constraints were suppressing learning on unlearnable examples, relaxing or removing them should disproportionately benefit the unlearnable group, narrowing the gap with the learnable group.

Results. Figure 5 shows neither intervention changes the training dynamics on unlearnable examples. The gap between learnable and unlearnable curves persists at essentially the same magnitude as in the baseline. This rules out gradient regularization as the cause of unlearnability.

Gradient Interference Analysis (Hypothesis 3)

Motivation. A third optimization-side hypothesis (Hypothesis 3, Appendix A.3) is gradient interference: within a batch, gradients from correct and incorrect rollouts for the same example might partially cancel each other if the rollouts share critical tokens. This could happen because GRPO assigns token-level gradients using a rollout-level advantage — every token in a correct rollout gets a positive gradient signal, and every token in an incorrect rollout gets a negative one. If correct and incorrect rollouts share many tokens (e.g., near-identical reasoning traces that diverge only at the final answer), the positive and negative gradients at those shared tokens cancel, leaving little net learning signal.

Measurement approach. The paper computes cosine similarity between gradient vectors of correct and incorrect rollouts at two levels:

Within-prompt: For each example, compute the gradient from a correct rollout and the gradient from an incorrect rollout (averaged over all correct-incorrect pairs), then measure their cosine similarity. A cosine similarity < 0 indicates the gradients point in opposite directions (interference); a similarity > 0 indicates they point in similar directions (no interference — they reinforce each other or are orthogonal).
Cross-prompt (dataset-level): For each group (unlearnable, learnable, easy), average the gradient of all correct rollouts in that group and compute cosine similarity with the average gradient of all incorrect rollouts across the entire training dataset.

Gradient computation details. For each example, the paper samples 100 examples from each group and 1000 rollouts per example under the initial policy, filters for correct rollouts, and computes the GRPO loss gradient following Equation 1. The per-rollout gradient is averaged first across tokens within the response and then across responses, yielding one gradient vector per example. For computational efficiency, gradients are computed with respect to a fixed, randomly initialized LoRA adapter attached to the base model rather than all parameters. The paper verifies that LoRA-based gradient similarity is highly correlated with full-parameter gradient similarity.

What cosine similarity means in this context: The cosine similarity between gradient vectors $g_1$ and $g_2$ is $\frac{g_1 \cdot g_2}{\|g_1\| \|g_2\|}$ , measuring the alignment of their directions in parameter space. A value of 1 means the gradients point in exactly the same direction (updating parameters in perfectly compatible ways); 0 means they are orthogonal (updating independent subspaces); -1 means they point in opposite directions (one gradient would undo the other's update).

Results. Figure 16a (within-prompt, at step 0): Both learnable and unlearnable groups show high positive cosine similarity (> 0.5) between correct and incorrect rollouts — their gradients are aligned, not opposing. At step 50 (mid-training), the distribution for learnable examples shifts — incorrect rollouts' gradients become negatively correlated with correct ones (mean near -0.2), similar to the easy group's pattern. This makes sense: as the model learns, the features that distinguish correct from incorrect solutions become more prominent in the gradient signal. However, the unlearnable group's distribution stays essentially unchanged — correct and incorrect rollouts remain aligned, meaning the model is not developing a representation that distinguishes them. Figure 16b (cross-prompt) shows no systematic difference between groups — gradient interference from incorrect rollouts in the full batch does not disproportionately affect unlearnable examples.

Conclusion: The failure to learn on unlearnable examples cannot be attributed to gradient cancellation between correct and incorrect rollouts. In fact, the persistently positive within-prompt similarity on unlearnable examples suggests a deeper issue — the model's gradients for correct and incorrect rollouts are indistinguishable, meaning the distinction between success and failure on these examples provides no useful optimization signal.

Cross-Example Gradient Similarity Analysis (Core Analytical Technique)

Motivation. After ruling out optimization-side explanations (Sections 4.1, 4.2, Appendix A.3), the paper turns to the hypothesis that unlearnability reflects a fundamental representation issue — specifically, that unlearnable examples are gradient outliers with learning signals that are poorly aligned with the rest of the training data (Section 5.1). If an example's gradient is very different from the average gradient of other examples, then the parameter updates that help other examples are unlikely to help this one, and vice versa — the example is isolated in optimization space.

Computation pipeline. For each group (unlearnable, learnable, easy), the paper:

Samples 100 examples from the group and 1000 rollouts per example under the initial policy (before any RL training).
Filters for rollouts with correct final answers.
Computes the GRPO loss gradient for each correct rollout following Equation 1, using the initial policy model.
Averages each rollout's gradient across tokens within the response, producing one gradient vector per rollout.
Averages across all correct rollouts for the same example, producing one gradient vector per example.
Computes pairwise cosine similarity between all example-level gradient vectors, yielding a similarity matrix.
For each example, computes its average similarity to all other examples in the dataset (across all three groups), producing a single scalar per example: how aligned this example's gradient is with the broader training distribution.

What this computes: For a given example $e$ with gradient vector $g_e$ (computed from its correct rollouts under the initial policy), the average similarity is:

$\text{sim}(e) = \frac{1}{N-1} \sum_{e' \neq e} \frac{g_e \cdot g_{e'}}{\|g_e\| \|g_{e'}\|}$

where $N$ is the total number of examples across all groups (300 — 100 from each group), $g_{e'}$ is the gradient vector for another example $e'$ , and $\frac{g_e \cdot g_{e'}}{\|g_e\| \|g_{e'}\|}$ is the cosine similarity between the two gradient vectors.

What the number means: A high average similarity (e.g., 0.7–0.8) means this example's gradient direction is strongly aligned with most other examples — parameter updates that help other examples will also help this one, and positive signals on this example provide reinforcement for what the model is learning elsewhere. A low similarity (e.g., 0.2–0.4) means this example's gradient points in a largely different direction from others — it is an outlier in optimization space, with learning signals that are mostly incompatible with the rest of the training distribution.

Why compare under the initial policy: Computing gradients before any RL training ensures the similarities reflect the model's initial representations, not artifacts of training. This is crucial for establishing causality — if unlearnable examples have low gradient similarity at initialization, this is a property of the pretrained model's representations, and RL training fails because it cannot reshape these representations. If the difference only appeared after training, it might be an effect of failed learning rather than its cause.

LoRA approximation. Computing full-parameter gradients for 100 examples × 1000 rollouts is computationally prohibitive. The paper attaches a fixed, randomly initialized LoRA adapter (a low-rank weight matrix added to selected layers) to the base model and computes gradients with respect to LoRA parameters only. The paper verifies that "LoRA-based gradient similarity is highly correlated with full-parameter gradient similarity," meaning the LoRA approximation preserves the relative ordering of similarities even if absolute values differ. This is critical for the validity of the comparison across groups — if LoRA similarity rankings were unrelated to full-parameter rankings, the analysis would be measuring an artifact.

Intra-group and inter-group analysis. Beyond individual examples, the paper computes the average similarity between groups (Figure 6). This reveals whether certain groups form distinct clusters in gradient space. The resulting $3 \times 3$ matrix shows:

Easy-easy similarity: $0.75 \pm 0.1$ — easy examples have highly consistent, concentrated gradients.
Learnable-learnable: $0.67 \pm 0.1$ — learnable examples are also fairly consistent but slightly more dispersed.
Unlearnable-unlearnable: $0.46 \pm 0.2$ — unlearnable examples have low similarity even with each other, with much higher variance.
Easy-learnable: $0.70 \pm 0.1$ — easy and learnable examples share strong gradient alignment.
Easy-unlearnable: $0.57 \pm 0.2$ — weak alignment with easy examples.
Learnable-unlearnable: $0.54 \pm 0.2$ — weak alignment with learnable examples.

What this pattern reveals: Easy examples form a tight cluster in gradient space — learning on one easy example strongly transfers to other easy examples, creating a virtuous cycle where each example reinforces the same update direction. Learnable examples are moderately aligned with this cluster, which is why they can be learned — they benefit from the momentum of the easy group while also contributing their own signals. Unlearnable examples, by contrast, are outliers not just from the main cluster but from each other — each unlearnable example is an island with its own idiosyncratic gradient direction. Unlike easy and learnable examples, which mutually reinforce, unlearnable examples cannot bootstrap each other because their gradients are inconsistent. And unlike learnable examples, they don't benefit from the easy group's gradient signal because the alignment is too weak.

Mid-training gradient similarity (Figure 11). The paper also computes gradient similarity at step 50 mid-training (Appendix A.4, Figure 17), showing that the overall gradient distribution spreads out as the model updates (expected — different examples' parameters move in different directions during training), but the relative ordering of groups persists: unlearnable examples still have lower similarity than learnable ones. This confirms that the gradient outlier property is persistent, not an initialization artifact that training would naturally resolve.

Relation to mid-training findings (Section 5.4). To demonstrate that gradient similarity is causally related to learnability (not just correlated), the paper examines the effect of mid-training on gradient alignment. Comparing Llama-3.2-3B-Base to OctoThinker-3B-Hybrid-Base and OctoThinker-3B-Long-Base (models mid-trained from Llama-3B on 20B tokens with different data mixtures; Wang et al., 2025), the paper computes gradient similarity between difficult examples and the broader set of 500 randomly sampled MATH Hard examples. Figure 11 shows both OctoThinker variants have consistently higher gradient similarity than the base Llama model. This means mid-training can increase gradient alignment — it reshapes representations so that initially outlier examples become better integrated with the training distribution. The key implication: unlearnability is fixable, but not during the RL stage; the representations must be aligned before RL begins.

Reasoning Quality Evaluation

Motivation. Gradient analysis reveals that unlearnable examples are outliers in optimization space, but doesn't explain why — what is different about these examples that causes such poor gradient alignment? Section 5.2 investigates this by examining the actual reasoning traces the model produces on unlearnable examples when it does (accidentally) get the correct final answer. The hypothesis is that these "correct" answers are often produced through flawed reasoning — the model arrives at the right number by coincidence or through shortcuts rather than genuine logical deduction, meaning the gradient signal it receives is based on an unreliable reasoning pattern that doesn't generalize.

Annotation methodology. For each group, the paper randomly samples 100 examples and gathers all rollouts with correct final answers (sampled from the initial policy before RL training). Each correct-answer rollout is evaluated by prompting GPT-5-mini with the problem statement and the model's reasoning trace, asking it to assign an integer score from 0 to 5:

0: Completely wrong reasoning (though the final answer happens to be correct).
5: Perfect, logically coherent reasoning.

The prompt specifically asks GPT-5-mini to judge whether the reasoning is "logically coherent and consistent," not whether the final answer is correct (since all evaluated rollouts already have correct answers). The output includes both a score and a text justification.

Why GPT-5-mini as evaluator: Using an LLM as a judge for reasoning quality is an imperfect but pragmatic choice — human annotation at scale for 300+ reasoning traces would be extremely expensive. GPT-5-mini's scores serve as a noisy but directionally informative proxy. The key comparisons are relative (across groups at the same time point, or within the same group across time), which controls for systematic biases in GPT-5-mini's scoring.

Results (Figure 1d and Figure 7). Figure 1d shows the distribution of quality scores at initialization:

Easy examples: high quality, tight distribution centered around 4–5.
Learnable examples: intermediate quality, broader distribution around 3–4.
Unlearnable examples: low quality, broad distribution centered around 2–3 with many scores of 0–1.

Figure 7 shows quality scores at steps 50 and 120 of RL training:

At step 50, the easy group's quality remains high, the learnable group has improved substantially (distribution shifted toward 4–5), but the unlearnable group has barely budged.
At step 120, the gap has widened further — the learnable group continues improving while the unlearnable group remains stagnant.

What this reveals: The model's ability to produce correct answers on unlearnable examples relies on "flawed" reasoning — internally inconsistent, sometimes containing serious errors (as in Table 2's example, where the reasoning contradicts itself and includes hallucinated cases). These correct answers don't represent genuine understanding; they represent lucky coincidences where multiple errors cancel out or where the model pattern-matches to a correct numeric answer without going through valid intermediate steps. Because the reasoning is unreliable, the gradient signal from these "correct" rollouts doesn't encode generalizable problem-solving patterns — it reinforces whatever heuristics (possibly including the errors) happened to produce the lucky answer. This connects directly to the gradient similarity finding: if the reasoning traces on unlearnable examples are incoherent, the gradient directions they produce will be inconsistent with each other (low intra-group similarity) and with examples that have coherent reasoning (low inter-group similarity).

Connection to the "bag of heuristics" hypothesis. Nikankin et al. (2025) showed that LLMs often solve math problems using "a bag of heuristics" — shallow pattern-matching strategies rather than algorithmic reasoning. The reasoning quality analysis provides direct evidence that unlearnable examples represent a subset where this heuristic-based solving is particularly unreliable. The model is not actually "reasoning" on these examples; it's guessing based on surface-level patterns, and its guesses are sometimes (but unreliably) correct. Binary outcome reward cannot distinguish these lucky guesses from genuine reasoning, so it reinforces both equally — but the gradient signal from lucky guesses is noisy and inconsistent, preventing the kind of coherent learning that happens on examples where the model can (or learns to) produce logically sound reasoning.

Data Augmentation Interventions

Motivation. If unlearnable examples have flawed representations (low gradient similarity, poor reasoning quality), can we fix this by providing additional training data that is semantically similar and thus should help the model develop better representations? Section 5.3 tests this by generating augmented examples that share the same problem-solving strategies as the unlearnable ones, and training on the augmented data jointly with the originals.

Two types of augmented data. For each unlearnable example, the paper generates two kinds of augmented problems using GPT-5 (with answers and step-by-step solutions provided by GPT-5, then cross-validated by Gemini-2.5-pro for correctness; Appendix C):

Similar problems ( $\mathcal{D}^{\text{sim}}_u$ ): Five problems per example that "test the same core skills and reasoning patterns" but "use different contexts, numbers, or scenarios." These are designed to be structural clones — same type of problem, different surface form.
Subproblems ( $\mathcal{D}^{\text{sub}}_u$ ): Decompositions of the original problem into independent subproblems whose solutions collectively solve the original. Each subproblem must be "fully self-contained," "unambiguous," and "have a unique, well-defined answer."

Why these two types: Similar problems test whether adjacent training examples can bootstrap better representations through shared problem structure — the idea is that seeing multiple variations of the same concept helps the model extract the invariant reasoning pattern. Subproblems test a compositionality hypothesis — if the model can learn to solve the subproblems independently, it might learn to compose those solutions to handle the full problem, even if the full problem initially had poor representations. The subproblem approach is more ambitious (and closer to curriculum learning) because it requires the model to learn new component skills.

Training configurations. The paper trains with three augmented data compositions using GRPO (same hyperparameters as baseline):

$\mathcal{D}_u + \mathcal{D}^{\text{sim}}_u$ : original unlearnable examples + similar problems.
$\mathcal{D}_u + \mathcal{D}^{\text{sub}}_u$ : original unlearnable examples + subproblems.
$\mathcal{D}_u + \mathcal{D}^{\text{sim}}_u + \mathcal{D}^{\text{sub}}_u$ : combined augmented set.

For comparative analysis, the same augmentation pipeline is also run on 100 randomly sampled learnable examples.

Results (Figure 8). The training reward on unlearnable examples shows at most marginal improvement under any augmented configuration — the curves are essentially flat regardless of what additional data is provided. The pass@1 and pass@16 on the validation set for the combined setting saturate very early (around 20–25 steps) and then decline, indicating overfitting without genuine learning. Figure 9 reveals a striking pattern: while the augmented subproblems ( $\mathcal{D}^{\text{sub}}_u$ , the green curve) show even better learnability than the original learnable examples (the orange curve — the model learns the subproblems smoothly and effectively), this learning does not transfer to the original unlearnable examples (the red curve, which stays flat). The model can learn to solve the decomposed subproblems but cannot compose those solutions to handle the full unlearnable examples.

Gradient similarity with augmented data (Figure 10). For both unlearnable and learnable examples, the paper plots each example's gradient similarity with the original training distribution against its similarity with its corresponding augmented similar problems. For the learnable group (Figure 10b), gradient similarity with augmented data is substantially higher (points cluster around 0.4–0.6) than for the unlearnable group (points cluster around 0.2–0.4 in Figure 10a), and there is essentially no correlation between the two measures (R² = 0.090). For the unlearnable group, there is a strong correlation (R² = 0.704) — examples that are outliers from the original distribution are also outliers from the augmented distribution.

What this means: Even semantically near-identical problems (same strategy, different numbers) have very different gradients when the original example is unlearnable. This is a profound finding — it means gradient similarity is not just about problem structure; it's about how the pretrained model's representations process that structure. The augmented similar problems are well-represented (the model can learn them), but the original unlearnable examples have some idiosyncratic property that causes the model to represent them differently — and that difference makes the gradient signals from the augmented data incompatible with learning the original examples. Data augmentation fails not because the augmented data is low-quality (it isn't — the model learns it smoothly), but because gradient transfer is blocked by the model's flawed internal representations of the original examples.

Why subproblems don't transfer either. The failure of subproblem learning to transfer is similarly informative. If the model can learn to solve subproblems A, B, C but cannot compose them to solve the full problem (A + B + C), this suggests the model hasn't learned to compose the skills — the subproblem skills exist as isolated islands in the model's parameter space, without the connective structure needed to chain them together. This aligns with findings from Yuan et al. (2025) and Park et al. (2025) that RL teaches composition of existing skills but cannot teach composition from scratch for examples where the pretrained model lacks the necessary representation structure.

Mid-Training Gradient Similarity Analysis

Motivation. Section 5.4 provides the positive contrast to the failure of RL-stage interventions — if representation flaws cause unlearnability, is there any stage of training where these flaws can be addressed? The paper examines mid-training, a stage between pretraining and RL where the model is further trained on domain-relevant data (typically using next-token prediction) to "warm up" the model for the RL stage.

Models compared. The paper analyzes three models:

Llama-3.2-3B-Base: The original pretrained model (no mid-training).
OctoThinker-3B-Hybrid-Base: Mid-trained from Llama-3B on 20B tokens with a hybrid data mixture.
OctoThinker-3B-Long-Base: Mid-trained from Llama-3B on 20B tokens with a different, longer-context data mixture (both from Wang et al., 2025).

Measurement. For each model, the paper randomly samples 500 examples from the MATH Hard training set and identifies a subset of difficult examples (using the same initial-pass@1 criterion). It then computes the gradient similarity between each difficult example and the full set of 500 examples using the same LoRA-based gradient computation pipeline.

Results (Figure 11). Both mid-trained OctoThinker models exhibit consistently higher gradient similarity than the base Llama model, with the distribution visibly shifted rightward. The base Llama model has a broad distribution of similarities centered around 0.2–0.3; both OctoThinker models have distributions centered around 0.3–0.4 with higher tails.

What this demonstrates: Mid-training can reshape model representations to improve gradient alignment for difficult examples. This is the crucial positive result — representation flaws are not inherently unfixable; they can be addressed through continued pretraining on appropriate data. The finding explains why mid-training has been empirically observed to improve RL performance (Wang et al., 2025): by aligning difficult examples' representations with the broader distribution, mid-training makes these examples more learnable when RL begins.

Implications for training pipelines. This result suggests a clear division of labor: pretraining and mid-training are responsible for ensuring examples have well-aligned representations (high gradient similarity), while RL's role is to amplify correct reasoning patterns within that aligned representation space. For examples with fundamentally misaligned representations, no amount of RL — regardless of sampling budget, algorithm improvements, or data augmentation — will help, because the gradient signals from those examples cannot transfer to or from the rest of the distribution. The practical recommendation is to invest more in mid-training to increase gradient alignment before RL, rather than trying to fix representation issues during the RL stage itself.

Curriculum Learning Experiment

Motivation. Appendix A.5 tests whether curriculum learning — training on easy and learnable examples first, then introducing unlearnable ones — can provide a scaffold for learning. The idea is that by the time the model sees unlearnable examples, it has already developed strong representations on related (easier) problems, and this prior knowledge might transfer.

Two-stage design:

Stage 1: Train the model on the full training data excluding unlearnable examples, until convergence.
Stage 2: Construct a new training set by randomly sampling the same number of examples as the unlearnable group from the Stage 1 data, and combining them with the unlearnable examples. Continue training on this mixed set.

Why this design: Stage 1 ensures the model has fully internalized the learnable and easy data before encountering unlearnable examples. Stage 2 then introduces unlearnable examples in a batch that is half learnable/easy (providing strong, aligned gradient signals) and half unlearnable — the idea being that the strong signals from the learnable half might provide some transfer benefit to the unlearnable half through parameter sharing.

Results (Figure 18). Figure 18a shows training reward on unlearnable examples over the epochs they are directly trained on (the x-axis counts only epochs of Stage 2). The baseline (training on the full dataset from the start) and the curriculum approach produce essentially identical flat reward curves — no improvement in either case. Figure 18b shows reasoning quality after Stage 1 of curriculum training: while easy and learnable examples have improved quality, unlearnable examples (which weren't trained on in Stage 1, so this reflects only indirect transfer) remain low. The paper also notes an "obvious drop in validation accuracy" when switching to Stage 2, suggesting that introducing unlearnable examples may actually degrade performance on other data.

Conclusion: Curriculum learning fails for the same fundamental reason as data augmentation — the representations on unlearnable examples are simply too poorly aligned for transfer from other examples, even after the model has become substantially better at related problems. The skills the model acquires in Stage 1 exist in a representation space that the unlearnable examples don't share, so those skills don't help.

Summary of Design Choices and Justifications

Intersection across three training runs for example categorization: Reduces noise from training stochasticity; an example must consistently fail (or succeed) to be classified. This conservative criterion likely underestimates the true fraction of unlearnable examples but ensures the analyzed subset is clean.
$\tau=0.1$ threshold and $N=32$ estimation: A practical operationalization that provides enough statistical reliability (32 samples gives reasonable pass@1 estimates) while keeping the experimental cost manageable. The exact threshold is acknowledged as "rather arbitrary" — the key findings are about distributional differences between groups, not the precise boundary placement.
LoRA-based gradient approximation: Necessary for computational feasibility (full-parameter gradients for 300,000+ rollouts would be impossible at this scale). The paper validates that LoRA similarity correlates with full-parameter similarity, making the relative comparisons valid even if absolute values differ.
GPT-5-mini for reasoning quality: An LLM judge is imperfect but enables annotation at scale. The relative comparisons (across groups, across time) control for systematic GPT biases, since any bias would affect all groups equally.
$4k$ oversampling in the replay experiment: The factor of 4 provides enough sampling to find $k_{\text{pos}}$ positive rollouts for most examples without requiring excessive compute. Smaller oversampling would make the replay buffer necessary for more examples, diluting the intervention.
Controlled gradient computation under the initial policy: Ensures that differences in gradient similarity are properties of the pretrained model, not effects of RL training. This is essential for establishing the causal direction — representation flaws precede and cause unlearnability, rather than resulting from it.
Data augmentation with GPT-5 + Gemini-2.5-pro validation: Ensures augmented problems are correct and genuinely similar to the originals, ruling out data quality as an explanation for augmentation's failure. The cross-validation by a different model (Gemini) adds a correctness check independent of the generation model.
Fixed, randomly initialized LoRA adapter: Using a fixed adapter (not one that changes during training) ensures that gradient similarity measurements at different training steps are comparable — if the adapter changed, differences in similarity could reflect adapter changes rather than model changes.
Edge case handling in the replay buffer: Limiting each buffered rollout to at most 2 replays prevents the same few rollouts from dominating gradient updates and creating a degenerate optimization loop.
Exclusion of zero-positive-reward examples from the "unlearnable" category: This is a crucial conceptual choice — the paper distinguishes "unlearnable" (correct rollouts exist but aren't learned from) from "currently unsolvable" (no correct rollouts exist), and focuses on the former as the more puzzling and theoretically informative phenomenon.

4. Key Insights and Innovations

Innovation 1: The Unlearnability Phenomenon as a First-Class Failure Mode Distinct from Exploration Bottlenecks

The paper's most fundamental conceptual contribution is the identification and rigorous operationalization of unlearnability as a distinct category of training failure that is categorically different from the exploration and reward-sparsity problems that dominate the RLVR literature. This is not an incremental empirical observation — it is a reframing of what the central obstacle to RL-based reasoning improvement actually is.

What the field assumed before this work. The implicit model underlying GRPO and related algorithms is that learning is bottlenecked by exploration: if the model can sample a correct trajectory, the contrastive signal between that trajectory and incorrect ones will steer the policy toward correctness. Under this model, any example where pass@k > 0 (i.e., correct rollouts exist) is in principle learnable, and the limiting factor is simply whether those rollouts occur frequently enough during training to provide sufficient gradient signal. This assumption motivated a substantial line of work on improving exploration — oversampling difficult prompts (Yu et al., 2025), entropy-based exploration bonuses (Cui et al., 2025; Cheng et al., 2025), fine-grained credit assignment (Liu et al., 2025b), and dynamic sampling schedules (Shi et al., 2025; Gao et al., 2025). It also motivated Sun et al. (2025a) to focus on "extremely difficult" examples with zero initial pass@k, showing that rich reward signals beyond binary correctness can make even these examples learnable. The shared assumption across all this work is a monotonic relationship: more (or better) reward signal → more learning.

What this paper shows instead. The unlearnability phenomenon breaks this monotonic relationship. Table 1 demonstrates that 16–30% of difficult examples (which do produce correct rollouts during training) remain flatlined throughout GRPO training across three model families and two datasets. The oversampling-with-replay experiment (Section 4.1, Figure 2) is the decisive test: it guarantees a fixed 1:7 ratio of positive to negative rollouts per example per batch, removing reward scarcity as a confound, yet the unlearnable group shows zero improvement. Even stronger interventions — SFT on distilled correct responses and RL with k=64 rollouts on unlearnable examples alone (Appendix A.2, Figure 15) — fail to close the gap. These results are not explainable by any theory that treats reward availability as the primary bottleneck.

Why this is a fundamental conceptual shift. This finding introduces a three-way taxonomy of training examples that the field previously lacked: easy (already solved), learnable (solved during RL), and unlearnable (never solved despite opportunities). Before this work, the implicit taxonomy was binary — solved vs. unsolved — with the assumption that unsolved examples could eventually be solved given enough compute and exploration. The binary view made it natural to treat all unsolved examples as a homogeneous category needing more exploration. The three-way view reveals that unsolved examples are heterogeneous in learnability, with a substantial subset that is inherently resistant to RL regardless of exploration budget. This resistance traces not to the RL algorithm's limitations but to properties of the examples themselves interacting with the pretrained model's representations (as the gradient analysis demonstrates). The implication is that improving RL algorithms — better exploration, finer-grained credit assignment, dynamic curriculum — can only help the learnable subset. For the unlearnable subset, the bottleneck is upstream of RL entirely.

The diagnostic value of this taxonomy. The paper's categorization methodology — initial pass@1 thresholding, reward trajectory analysis, and intersection across three independent training runs — provides a reusable diagnostic tool for any RLVR training pipeline. Rather than treating low-reward examples as a homogeneous "needs more training" category, practitioners can now distinguish between "genuinely learnable" examples (where more compute or better exploration might help) and "genuinely unlearnable" examples (where no amount of RL will work and resources should be redirected to pretraining or mid-training). This is a practical advance with immediate implications for data selection, curriculum design, and compute allocation.

Evidence anchoring. The core numbers are in Table 1 — unlearnable examples constitute 16.7–30.2% of difficult examples across settings. The critical evidence that this is not an exploration artifact is Figure 2 (oversampling with replay fails to help) and Figure 15 (SFT and large-k RL fail to help). The cross-model and cross-dataset consistency of the phenomenon (Table 1) establishes that unlearnability is not an idiosyncratic property of a particular model or training run.

Innovation 2: Gradient Similarity as a Causal Diagnostic for Learnability, Not Just a Correlational Metric

The paper's second major conceptual contribution is repurposing cross-example gradient similarity from a descriptive metric to a causal diagnostic that predicts learnability and reveals the mechanism of failure. This is not incremental — gradient similarity has been used before as a descriptive tool for understanding neural network training dynamics, but the paper deploys it in a novel way that establishes a causal chain: (1) low gradient similarity at initialization → (2) isolation in optimization space during RL → (3) failure to learn despite positive reward signals → (4) ungeneralizable reasoning patterns that the binary reward cannot distinguish from genuine reasoning.

What the field previously did with gradient similarity. Gradient similarity has a long history as a descriptive metric in deep learning — measuring how aligned updates from different examples are, characterizing interference vs. reinforcement in multi-task learning, diagnosing catastrophic forgetting, etc. In the RLVR literature specifically, gradient analysis has been used to study gradient interference between correct and incorrect rollouts (Nguyen et al., 2025), showing that when correct and incorrect rollouts share tokens, their gradients partially cancel. But this work treated gradient similarity as a symptom of optimization dynamics — something that emerges during training as a consequence of the learning algorithm and can potentially be mitigated by algorithmic changes.

What this paper does differently. The paper's key methodological move is computing gradient similarity under the initial policy before any RL training occurs (Section 5.1). This is critical because it reframes gradient similarity from a dynamic property of the training process to a static property of the pretrained model's representations. Under this reframing, gradient similarity becomes a diagnostic for representation quality, not optimization quality. The finding that unlearnable examples have substantially lower gradient similarity with the rest of the training distribution at initialization (Figure 1c, Figure 6) means that the representations are flawed before RL even begins — the model's internal processing of these examples produces gradient signals that are fundamentally misaligned with what it learns from other examples. RL doesn't cause this misalignment; it merely fails to overcome it.

Why this is causal, not just correlational. The paper supports the causal interpretation through three converging lines of evidence:

Temporal precedence: Low gradient similarity is measured at initialization, before any RL training, establishing that it precedes (and thus cannot be caused by) failed RL learning.
Intervention failure: The optimization-side interventions (Sections 4.1, 4.2, Appendix A.3) — oversampling, removing clipping, removing KL penalty, preventing gradient interference — all fail to make unlearnable examples learnable. If gradient similarity were merely a symptom of optimization failures, fixing those failures should increase both similarity and learnability. The fact that it doesn't means the low similarity reflects a deeper representation property that optimization interventions cannot change.
Positive intervention (mid-training): Section 5.4 shows that mid-training — an intervention at the pretraining stage that reshapes representations — can increase gradient similarity (Figure 11). This demonstrates that gradient similarity is malleable through representation-level interventions (pretraining/mid-training) but not through optimization-level interventions (RL), establishing a causal distinction between what gradient similarity measures (representation quality) and what RL can affect (policy optimization within an existing representation space).

The diagnostic power. This reframing has significant practical implications. Computing gradient similarity at initialization requires only the pretrained model and training data — no expensive RL training runs needed — yet it predicts which examples will be learnable under RL. This makes it a cheap, model-intrinsic diagnostic for data quality in the context of RLVR: examples with low gradient similarity to the training distribution are unlikely to benefit from RL regardless of compute budget, and should be either removed from the RL training set or addressed through pretraining/mid-training interventions before RL begins.

Contrast with prior gradient interference work. Nguyen et al. (2025) used gradient similarity to show that interference between correct and incorrect rollouts within the same batch limits learning — an optimization-side explanation. This paper's gradient analysis is fundamentally different in both method (computing similarity across examples using only correct rollouts, not within-example correct-vs-incorrect similarity) and implication (pointing to pretraining representation quality, not RL algorithm design). The two uses of gradient similarity are complementary but address different questions: Nguyen et al. ask "does the RL algorithm's gradient computation cause interference?" while this paper asks "are the pretrained representations of this example compatible with learning from other examples?"

Evidence anchoring. The core gradient similarity distributions are in Figure 1c (box plot showing unlearnable examples have markedly lower similarity than learnable and easy examples) and Figure 6 (the 3×3 inter-group similarity matrix, showing easy examples cluster tightly at 0.75, learnable at 0.67, and unlearnable at 0.46 with high variance). The mid-training result is Figure 11 (OctoThinker models show higher similarity than base Llama). The temporal analysis showing persistence of low similarity through training is Figure 17.

Innovation 3: The Representational "No-Man's Land" — Why RL Cannot Fix What Pretraining Broke

The third conceptual contribution is the paper's demonstration that there exists a regime of representation quality where examples are "good enough" to occasionally produce correct answers (triggering positive reward signals) but "bad enough" that those signals cannot be integrated into coherent learning. This is a nuanced boundary condition on RLVR's effectiveness that was previously unrecognized, and it explains why the optimization-side interventions fail so comprehensively.

The implicit assumption the paper overturns. The standard RLVR narrative assumes a smooth relationship between representation quality and learnability: better representations → more correct rollouts → stronger gradient signal → more learning. Under this view, the only problematic regime is when representations are so poor that correct rollouts never occur (pass@k ≈ 0), in which case there's simply no positive signal to learn from. The expectation is that once representations cross the threshold where correct rollouts begin appearing (pass@k > 0), learning should proceed smoothly, with the speed of learning proportional to the frequency of correct rollouts.

What the evidence shows instead. The unlearnability phenomenon reveals a discontinuity in this assumed smooth relationship. Unlearnable examples are in a peculiar intermediate regime: their representations are good enough that the model can occasionally produce the correct final answer (satisfying the binary verifier), but bad enough that the reasoning producing those correct answers is unreliable, incoherent, and ungeneralizable (Section 5.2, Figure 1d, Table 2). When a correct rollout occurs on such an example, the gradient signal it produces points in a largely idiosyncratic direction (low gradient similarity, Figure 6), encoding whatever unstable heuristic produced the lucky answer rather than any generalizable problem-solving pattern. The optimization algorithm dutifully follows this gradient, but because the direction is inconsistent both with other examples (low inter-group similarity) and with other correct rollouts on the same example (the example's internal gradient is not stable enough to produce a consistent direction across multiple lucky samples), the update produces no lasting improvement in the probability of correct answers on that example.

Why this is a "no-man's land." The representation quality of unlearnable examples sits in an uncomfortable intermediate zone:

Too good to be ignored: The examples produce correct rollouts, so they pass dynamic sampling filters (they contribute to gradient updates) and receive positive advantages. They cannot be trivially identified and excluded without some external criterion.
Too bad to be learned: The gradient signals from these correct rollouts are too misaligned with the broader training distribution to produce transferable learning, and too inconsistent with each other to produce self-reinforcing learning on the example itself.
Inaccessible to RL fixes: The optimization-side interventions tested in Section 4 (more positives, less clipping, no KL, curriculum) all operate within the existing representation space and cannot reshape the representations themselves. The representations are the problem, and RL can only optimize within them, not restructure them.

This intermediate zone is what makes unlearnability both practically important and theoretically interesting. If unlearnable examples were simply unsolvable (zero pass@k), they would be filtered by dynamic sampling and ignored — a non-issue. If they were easily solvable (high pass@k), they would saturate and also be filtered — also a non-issue. It is precisely because they sit in this middle ground — solvable enough to participate in training, but not solvable enough to benefit from it — that they create a persistent drain on training compute with no return.

Connection to the "bag of heuristics" hypothesis. This finding gives concrete empirical form to Nikankin et al.'s (2025) theoretical argument that LLMs solve math problems using shallow heuristics rather than algorithmic reasoning. Unlearnable examples appear to be those where the model's heuristic bag contains patterns that sometimes (but unreliably) produce the correct answer, but where no coherent reasoning structure exists to be reinforced. The binary reward cannot distinguish a lucky heuristic match from genuine reasoning, so it reinforces both — but the gradient from lucky heuristics is noisy, inconsistent, and ungeneralizable, while the gradient from genuine reasoning is structured and transferable. The unlearnability phenomenon is thus the failure mode of heuristic-based solving under outcome-only reward.

What this changes about how we think about RLVR data. The standard advice for RLVR data curation is to ensure correct answers exist in the model's output distribution (pass@k > 0) and that there is sufficient reward variance for contrastive learning (dynamic sampling). The unlearnability phenomenon adds a third, more stringent criterion: examples must not only have correct rollouts, but those correct rollouts must be produced through coherent reasoning that produces generalizable gradient signals. This is a fundamentally different requirement — it's about the quality of correct rollouts, not just their existence. It suggests that data curation for RLVR should include reasoning quality assessment (using an LLM judge or other automatic metric) to filter out examples where the model can produce correct answers only through unreliable heuristics, since these examples will contribute noisy gradients without improving.

Evidence anchoring. The reasoning quality distributions are in Figure 1d (initial) and Figure 7 (steps 50 and 120), with the persistent gap between unlearnable and learnable groups demonstrating that reasoning quality is both predictive of and resistant to improvement during RL. The concrete example of flawed reasoning is Table 2, with errors highlighted in red — this is a direct illustration of a "correct" answer produced through incoherent intermediate steps. The gradient similarity results that establish the mechanism are in Figures 1c and 6.

Innovation 4: Data Augmentation's Failure as Evidence for a Representation Barrier (a Strong Negative Result)

The paper's fourth conceptual contribution is a strong negative result that reveals the nature of the representation problem: semantically near-identical problems can have dramatically different gradient signatures when the original example is unlearnable, meaning the representational flaws are not about problem structure but about how the specific example is processed by the pretrained model.

The prior expectation. Data augmentation — generating variations of training examples — is a standard technique for improving model robustness and generalization across machine learning. In the context of RLVR, the natural hypothesis is that augmenting unlearnable examples with structurally similar problems should help by providing additional gradient signals with compatible optimization directions. The augmented problems test the same core skills, have near-identical structure (often differing only in numbers; see Tables 3 and 4), and are verified correct by a separate model. Under standard assumptions about neural network generalization, training on these augmented examples should transfer to the originals — the model should extract the invariant reasoning pattern and apply it to the unlearnable case.

What the experiment shows instead. The data augmentation results (Section 5.3, Figures 8 and 9) deliver a clean negative result at every level:

Training reward on unlearnable examples stays flat regardless of whether augmented data is included (Figure 8, left y-axis).
Pass@1 and pass@16 on validation data saturate early and then decline (Figure 8, right y-axis), indicating overfitting without genuine learning.
The augmented subproblems are learned smoothly (Figure 9, green curve) — the model genuinely acquires the ability to solve the decomposed subproblems — but this learning has zero transfer to the original unlearnable examples (red curve, Figure 9).
Gradient similarity with augmented data is strongly correlated with gradient similarity to the original distribution for unlearnable examples (R² = 0.704, Figure 10a), meaning that if the original example is a gradient outlier, the augmented versions of it are also outliers. This correlation is absent for learnable examples (R² = 0.090, Figure 10b), where augmented data's gradients are similar to the broader distribution regardless of the original's similarity.

Why this is a conceptually significant negative result. The failure of data augmentation is not just an empirical disappointment — it reveals a fundamental property of the representation problem that distinguishes it from standard generalization failures:

Not about data diversity: If the problem were simply insufficient coverage of the problem type, augmenting with similar problems would help. It doesn't.
Not about problem complexity: If the problem were that the original examples are too hard, decomposing them into subproblems and learning those subproblems would enable composition back to the full problem (as Yuan et al., 2025 and Park et al., 2025 found for other examples). The model can learn the subproblems but cannot compose them — suggesting the composition skill itself depends on representations that the unlearnable examples lack.
About example-specific representation flaws: The strong R² = 0.704 correlation in Figure 10a means that the gradient outlier property is intrinsic to each specific unlearnable example — even data generated to be structurally identical produces gradients that are similarly misaligned. This points to some example-specific feature (perhaps specific number combinations, specific phrasing patterns, or specific reasoning steps) that the pretrained model processes in an aberrant way, and that aberration persists across semantically identical variations.

The theoretical import. This result strengthens the paper's central claim that unlearnability is a representation-level problem. If the barrier were at the level of problem-solving strategy (the model doesn't know how to solve problems of this type), then teaching it the type through augmented examples should work. If the barrier were at the level of specific surface features (the model gets confused by particular numbers or phrasing), then varying those features through augmentation should work. The fact that neither works, combined with the gradient similarity evidence, indicates the barrier is at the level of how the pretrained model's internal representations process these specific examples — a level that is opaque to semantic analysis of the problem text and resistant to intervention through data manipulation at the RL stage.

Practical implications for data synthesis. The finding that semantically similar examples are not necessarily similar in optimization space (Figure 10) has important implications for the growing practice of synthetic data generation for LLM post-training. It suggests that data synthesis pipelines should not assume that structural similarity implies gradient similarity or transferability. Examples that are gradient outliers for the pretrained model may need to be identified and addressed through representation-level interventions (pretraining/mid-training on those specific examples or close variants) rather than through RL-stage data augmentation.

Evidence anchoring. The core results are Figures 8, 9, and 10. Tables 3 and 4 provide concrete examples of augmented problems that are structurally near-identical to the originals yet produce dissimilar gradients. The R² = 0.704 vs. 0.090 contrast in Figure 10 is the key statistical evidence distinguishing the behavior of unlearnable and learnable examples under augmentation.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses two primary datasets: (1) MATH (Hendrycks et al., 2021) — specifically, Qwen2.5-0.5B is trained on MATH difficulty levels 1–4 ("MATH Easy") and Llama-3.2-3B-Instruct on levels 3–5 ("MATH Hard"), following prior work (Zeng et al., 2025). Evaluation uses MATH 500, a 500-question subset. (2) DeepScaleR (Luo et al., 2025) — a large-scale dataset with 40k math problems and verifiable answers, used for Qwen2.5-3B training. The authors randomly split this into 90% training and 10% validation. All datasets provide ground-truth answers enabling binary outcome verification.
Base model(s). Three model configurations are tested: Qwen2.5-0.5B (Qwen et al., 2025) on MATH Easy, Qwen2.5-3B on DeepScaleR, and Llama-3.2-3B-Instruct (Grattafiori et al., 2024) on MATH Hard. These span two model families (Qwen and Llama), two scales (0.5B and 3B), and both base and instruct variants, providing evidence that the unlearnability phenomenon is robust across model architectures and training recipes. For the mid-training analysis (Section 5.4), the paper additionally evaluates OctoThinker-3B-Hybrid-Base and OctoThinker-3B-Long-Base (Wang et al., 2025), both mid-trained from Llama-3.2-3B-Base on 20B tokens with different data mixtures.
Metrics. The primary metrics throughout the paper are:
- Average training reward — the mean binary reward (1 if final answer matches ground truth, 0 otherwise) across rollouts for a given group of examples at each training step, computed from the original sampled rollouts before any replay or filtering. This tracks whether the model's probability of producing correct answers on a group of examples changes over the course of RL training.
- Pass@1 — the fraction of responses that produce the correct final answer when sampling one response per example from the policy model. For the unlearnability definition (Section 3.2), pass@1 is estimated with $N=32$ samples per example; the threshold for "unlearnable" is $\text{pass@1} < 0.1$ .
- Pass@k (specifically pass@16) — used in the data augmentation experiments (Figure 8) to assess whether the model can produce correct answers in at least one of 16 independent samples, providing a complementary view to pass@1.
- Gradient similarity — cosine similarity between example-level gradient vectors computed from correct rollouts under the initial policy (Section 5.1). This is the paper's core analytical metric for diagnosing representation alignment. Gradients are computed with respect to a fixed, randomly initialized LoRA adapter for computational efficiency; the paper verifies that LoRA-based similarity is highly correlated with full-parameter similarity.
- Clipping ratio — the fraction of tokens in a group's rollouts whose probability ratio $r_{i,t}$ falls outside the clipping range $[1-\varepsilon, 1+\varepsilon]$ during GRPO training (Figure 4), used to test whether gradient regularization disproportionately affects unlearnable examples.
- Reasoning quality — an integer score from 0 to 5 assigned by GPT-5-mini (Singh et al., 2025) to model-generated reasoning traces that reached correct final answers (Section 5.2). The prompt asks GPT-5-mini to judge "logical coherence and consistency." Quality is assessed at initialization and at multiple points during RL training (steps 50 and 120).
- Reference log-likelihood — $\log \pi_{\text{ref}}(y_{\text{correct}} | x)$ , the log-probability the reference (initial) policy assigns to a correct-answer rollout (Figure 3), used to test whether unlearnable examples have systematically lower-probability correct rollouts.
Baselines. The paper uses several baseline configurations and comparisons:
- Naive GRPO with dynamic sampling (Section 3.1) — the standard GRPO algorithm with DAPO-style filtering (Yu et al., 2025) that removes prompts with zero reward variance from gradient updates. This is the primary baseline against which all interventions are compared.
- GRPO with clip-higher — following DAPO (Yu et al., 2025), the clipping upper bound is raised so that positive-advantage tokens are not capped (Section 4.2, Figure 5). This tests whether standard PPO clipping suppresses learning on unlearnable examples.
- GRPO without KL loss — the KL penalty term $-\beta \text{KL}(\pi_\theta|\pi_{\text{ref}})$ is removed entirely (Section 4.2, Figure 5), testing whether reference-model constraints prevent learning.
- GRPO with oversampling and replay — a custom intervention guaranteeing exactly $k_{\text{pos}}=1$ correct rollout per batch per example (Section 4.1, Algorithm 1, Figure 2), used to test the positive-rollout-scarcity hypothesis.
- Supervised fine-tuning (SFT) on distilled correct responses — Qwen2.5-7B generates correct solutions on MATH training data, which are used to fine-tune Qwen2.5-0.5B (Appendix A.2, Figure 15a). This is a stronger-than-RLVR baseline testing whether direct supervision can overcome unlearnability.
- GRPO on unlearnable examples alone with $k=64$ — training only on unlearnable examples with a much larger rollout group (Appendix A.2, Figure 15b), testing whether simply providing more positive rollouts per gradient step helps.
- Curriculum learning — two-stage training: first on all data excluding unlearnable examples, then mixed training with unlearnable examples reintroduced (Appendix A.5, Figure 18). This tests whether prior learning on easier data scaffolds later learning on unlearnable data.
- Data augmentation variants — GRPO training on unlearnable examples combined with GPT-5-generated similar problems ( $\mathcal{D}_u + \mathcal{D}^{\text{sim}}_u$ ), subproblems ( $\mathcal{D}_u + \mathcal{D}^{\text{sub}}_u$ ), or both ( $\mathcal{D}_u + \mathcal{D}^{\text{sim}}_u + \mathcal{D}^{\text{sub}}_u$ ) (Section 5.3, Figures 8, 9, 10). These test whether semantically similar training data can improve representations and learning on unlearnable examples.
- Mid-training comparison — gradient similarity is compared between base Llama-3.2-3B-Base and two OctoThinker variants mid-trained on 20B tokens (Section 5.4, Figure 11). This tests whether pretraining-stage interventions can improve gradient alignment for difficult examples.
Generation budget / compute accounting. The paper measures test-time compute in terms of the number of rollouts sampled per example per training step. Standard GRPO uses $k=8$ rollouts per prompt. The oversampling-with-replay experiment (Section 4.1) samples $4k = 32$ rollouts per example, then downsamples to $k=8$ while enforcing the $k_{\text{pos}}=1$ constraint. The large-k experiment (Appendix A.2) uses $k=64$ . For gradient computation (Section 5.1), 1000 rollouts per example are sampled under the initial policy. Training compute is not directly compared across interventions — each intervention is evaluated by its effect on training reward curves and final pass@1 relative to the naive GRPO baseline at the same training steps.
Cross-validation / statistical protocol. The paper employs several noise-reduction techniques:
- Intersection across three independent training runs: For categorizing examples as unlearnable or learnable (Section 3.2), the authors run GRPO training three times independently for each model-setting combination. An example is only classified as unlearnable if it falls below the $\tau=0.1$ pass@1 threshold in all three runs; similarly, learnable examples must exceed the threshold in all three runs. This conservative criterion reduces classification noise from training stochasticity. For examples with no positive reward, the union across three runs is taken (an example is excluded if it never received positive reward in any run).
- Gradient similarity computation under initial policy: All gradient similarity measurements (Figures 1c, 6, 10, 11, 17) are computed using the pretrained model before any RL training, ensuring that differences reflect properties of the initial representations rather than artifacts of training dynamics.
- GPT-5-mini scoring with justification: Reasoning quality annotations (Section 5.2) include both a numeric score and a text justification, enabling qualitative validation of the scores. The use of an LLM judge for relative comparisons (across groups, across time) controls for systematic biases since any bias would presumably affect all groups equally.
- Data augmentation cross-validation: Augmented problems generated by GPT-5 are cross-validated by Gemini-2.5-pro for correctness before being used in training (Section 5.3, Appendix C), ensuring augmented data quality does not confound the results.

Main Quantitative Results

Prevalence of Unlearnable Examples Across Models and Datasets

The paper's first empirical finding is that unlearnable examples constitute a substantial fraction of the difficult training set across all tested configurations (Table 1). After excluding examples that never observe any positive reward during training:

Qwen2.5-0.5B on MATH Easy: 30.2% of difficult examples are unlearnable, 25.6% are learnable, and 23.5% never receive positive reward (excluded from both categories). Among examples that do receive positive reward, 54% are unlearnable and 46% are learnable — roughly half.
Llama-3.2-3B-Instruct on MATH Hard: 21.9% unlearnable, 31.6% learnable, 37.7% without positive reward. Among positive-reward examples, 41% are unlearnable.
Qwen2.5-3B on DeepScaleR: 16.7% unlearnable, 14.2% learnable, 47.2% without positive reward. Among positive-reward examples, 54% are unlearnable.

The key observation is that across all settings, after excluding examples that never see correct rollouts, approximately half of the remaining difficult examples are unlearnable. This establishes that the unlearnability phenomenon is not a rare edge case — it affects a substantial portion of the training distribution regardless of model family (Qwen vs. Llama), model scale (0.5B vs. 3B), or dataset (MATH vs. DeepScaleR). The high percentages of examples without positive reward (23.5–47.2%) also indicate that a large fraction of difficult examples are rarely or never solved correctly by the initial policy, but these are excluded from the unlearnability analysis because their failure can be trivially attributed to absence of positive signal — the unlearnable subset is specifically those that do receive positive signals yet still fail to learn.

Positive Rollout Scarcity Does Not Explain Unlearnability

The oversampling-with-replay experiment (Section 4.1, Figure 2) directly tests Hypothesis 1 — that unlearnable examples fail because they see too few correct rollouts. The intervention guarantees exactly $k_{\text{pos}}=1$ positive rollout per batch per example for both learnable and unlearnable groups. The result has two components:

Effect on learnable examples: The training reward on the learnable group rises more slowly under oversampling with replay than under naive GRPO (the orange dashed curve in Figure 2 is below the orange solid curve for most of training). This is expected — the intervention constrains the number of correct rollouts to exactly one per batch even when the model would naturally produce more correct rollouts as it improves, reducing the effective reward variance and slowing learning.
Effect on unlearnable examples: The training reward on the unlearnable group remains essentially flat under both naive GRPO and oversampling with replay (the blue solid and dashed curves in Figure 2 are nearly identical and near zero throughout training). Guaranteeing a positive rollout in every batch provides no benefit.

The gap between learnable and unlearnable curves persists under the controlled condition. This is the critical evidence ruling out Hypothesis 1 — if reward scarcity were the cause, equalizing the number of positives should narrow or close the gap. It does not.

Appendix A.2 provides two additional tests that strengthen this conclusion:

SFT on distilled correct responses (Figure 15a): Even direct supervised fine-tuning on correct solutions generated by Qwen2.5-7B fails to substantially improve pass rate on unlearnable examples, while learnable and easy examples improve as expected. This is a much stronger intervention than RL — the model sees ground-truth correct reasoning, not just binary rewards — yet unlearnable examples remain resistant.
Large-k RL on unlearnable examples alone (Figure 15b): Training GRPO with $k=64$ rollouts per prompt exclusively on unlearnable examples — a setting with many more correct rollouts per gradient step and no interference from other data — produces no meaningful improvement in average training reward. The curve stays flat near zero despite many training epochs.

Together, these three experiments converge on a strong negative result: increasing the quantity or density of positive supervision (through more rollouts, enforced ratios, or direct SFT) does not make unlearnable examples learnable. The bottleneck is not reward availability.

Gradient Regularization Does Not Suppress Learning on Unlearnable Examples

Section 4.2 tests Hypothesis 2 — that clipping and KL penalties disproportionately suppress gradients from unlearnable examples' correct rollouts. The evidence against this hypothesis is threefold:

Reference log-likelihood distributions are comparable (Figure 3). The distributions of $\log \pi_{\text{ref}}(y_{\text{correct}} | x)$ — how probable the reference model considers correct-answer rollouts — are plotted as violin plots for unlearnable, learnable, and easy groups. The distributions substantially overlap, with no systematic shift toward lower values for the unlearnable group. This directly contradicts the mechanism proposed in Hypothesis 2: if correct rollouts on unlearnable examples were low-probability under the reference policy, their probability ratios $r_{i,t}$ would be extreme, leading to more clipping. The data show this is not the case.

Clipping rates track closely across groups (Figure 4). The fraction of tokens whose probability ratio is clipped is plotted as a function of training steps for all three groups. All three curves hover in the range of 0.1–0.5% throughout training and follow nearly identical trajectories. Unlearnable examples are not clipped more frequently than learnable or easy examples.

Ablating clipping and KL penalty does not help (Figure 5). Two interventions are tested:

Clip-higher: Raising the upper clipping bound so that positive-advantage tokens are never capped. The training reward curves for both learnable and unlearnable groups (dashed lines) are nearly identical to the baseline GRPO curves (solid lines) — the gap between learnable and unlearnable persists unchanged.
No KL loss: Removing the KL penalty term entirely. Again, the curves (dotted lines) show no improvement over baseline for either group.

These results rule out Hypothesis 2 comprehensively. Unlearnable examples are not edge cases where the optimizer's regularization mechanisms happen to suppress the beneficial gradient — their resistance to learning stems from factors orthogonal to the probability ratios and reference-model constraints.

Gradient Interference Does Not Explain Unlearnability

Appendix A.3 tests Hypothesis 3 — that gradients from correct and incorrect rollouts within the same prompt partially cancel because they share critical tokens, leaving little net learning signal on unlearnable examples. The evidence comes from cosine similarity between correct-rollout gradients and incorrect-rollout gradients at two levels:

Within-prompt similarity (Figure 16a). At step 0 (initialization), both learnable and unlearnable groups show high positive cosine similarity (> 0.5) between correct and incorrect rollouts — their gradients are aligned, not opposing. At step 50 (mid-training), the distribution for learnable examples shifts toward negative values (mean near −0.2), similar to the easy group — the model has learned representations where correct and incorrect outputs produce gradients that point in different directions, enabling contrastive learning. The distribution for unlearnable examples, however, remains essentially unchanged — correct and incorrect rollouts stay aligned, meaning the model is not developing representations that distinguish success from failure on these examples. This is the opposite of what gradient interference would predict: interference would cause negative similarity, but unlearnable examples show persistently positive similarity.

Cross-prompt similarity (Figure 16b). Cosine similarity between each group's correct-rollout gradients and all incorrect-rollout gradients across the dataset shows no systematic difference between groups. Gradient interference from incorrect rollouts in the full batch does not disproportionately affect unlearnable examples.

The takeaway is not that interference explains unlearnability, but rather that the absence of a developing distinction between correct and incorrect rollouts on unlearnable examples indicates a deeper representation problem — the model's internal processing of these examples fails to produce features that differentiate good from bad solutions, making contrastive learning impossible regardless of the number of correct or incorrect rollouts.

Unlearnable Examples Are Gradient Outliers

The cross-example gradient similarity analysis (Section 5.1) is the paper's central empirical finding connecting the optimization failures to representation quality. The key results are:

Gradient similarity distribution by group (Figure 1c). Computing each example's average cosine similarity to all other examples in the training set (300 total: 100 unlearnable, 100 learnable, 100 easy) using gradients from correct rollouts under the initial policy, the distributions show:

Easy examples: concentrated at high similarity (predominantly 0.6–0.9).
Learnable examples: intermediate, slightly broader distribution (predominantly 0.5–0.8).
Unlearnable examples: substantially lower, with a distribution centered around 0.3–0.6 and a long tail extending from 0 to 0.4.

The visual separation is clear — unlearnable examples as a group have markedly lower gradient alignment with the broader training distribution.

Inter-group and intra-group similarity matrix (Figure 6). The 3×3 matrix of average cosine similarities (mean ± standard deviation) reveals the structure of the gradient space:

From \ To	Unlearnable	Learnable	Easy
Unlearnable	0.46 ± 0.2	0.54 ± 0.2	0.57 ± 0.2
Learnable	0.54 ± 0.2	0.67 ± 0.1	0.70 ± 0.1
Easy	0.57 ± 0.2	0.70 ± 0.1	0.75 ± 0.1

Several patterns are notable:

Unlearnable-unlearnable similarity (0.46) is the lowest of all pairwise comparisons and has the highest variance (±0.2). Unlearnable examples are not just outliers from the main distribution — they are outliers from each other. Each unlearnable example has its own idiosyncratic gradient direction, meaning they cannot mutually reinforce through shared gradient signals.
Easy-easy similarity (0.75) is the highest, with low variance (±0.1). Easy examples form a tight cluster in gradient space — learning on one easy example strongly transfers to others.
Easy-learnable similarity (0.70) is high, explaining why learnable examples can be learned — they benefit from gradient alignment with the easy group.
Unlearnable-learnable (0.54) and unlearnable-easy (0.57) are much lower than the within-learnable/easy similarities. The gradient transfer from the main cluster to unlearnable examples is weak.

Mid-training similarity (Figure 17). Gradient similarity computed at step 50 of training shows the overall distribution spreading out (expected as the model updates), but the relative ordering persists — unlearnable examples still have lower similarity than learnable ones. This confirms that the gradient outlier property is not resolved by RL training itself.

These results establish that unlearnable examples have fundamentally different gradient signatures from the rest of the training distribution at initialization, before RL training begins. This is the core evidence for the paper's representation-quality hypothesis — the flaws that cause unlearnability exist in the pretrained model's representations, not in the RL optimization process.

Unlearnable Examples Have Persistently Low Reasoning Quality

Section 5.2 connects the gradient outlier finding to observable behavior by analyzing the reasoning traces of correct-answer rollouts. The key results:

Initial reasoning quality (Figure 1d). At initialization, GPT-5-mini scores for correct-answer reasoning traces show:

Easy examples: predominantly scores of 4–5, tight distribution.
Learnable examples: scores clustered around 3–4, broader distribution.
Unlearnable examples: scores spread across 0–3, with many examples scoring 0–1 — these are correct answers reached through seriously flawed or incoherent reasoning.

The concrete example in Table 2 illustrates what "low-quality reasoning" means: the model makes a correct initial observation, then introduces nonsensical cases ("Case 2: |x + y + z| = 0" when the problem does not allow this), contradicts its own earlier statements, and arrives at the correct answer essentially by accident. The reasoning is not logically coherent but the final answer happens to match.

Reasoning quality trajectory during RL training (Figure 7). At step 50 (Figure 7a), the learnable group's reasoning quality has improved substantially (distribution shifted toward 4–5, approaching easy-group levels), while the unlearnable group's quality has barely budged — it remains concentrated at low scores. At step 120 (Figure 7b), the learnable group continues improving and the gap with unlearnable examples has widened further.

Curriculum learning does not transfer reasoning quality (Figure 18b). After Stage 1 of curriculum learning (training only on easy and learnable examples), reasoning quality on unlearnable examples — which were excluded from Stage 1 training — remains low. The skills the model acquires on learnable data do not transfer to improve reasoning quality on unlearnable examples.

These findings provide a behavioral counterpart to the gradient similarity results. The gradient outlier property manifests as unreliable, incoherent reasoning on unlearnable examples — the model can sometimes produce the correct final answer, but through processes that are not generalizable and that produce inconsistent gradient signals. RL, which only sees the final answer correctness, reinforces whatever heuristic produced the lucky answer, but because that heuristic is different each time (and different from the coherent reasoning patterns used on easy/learnable examples), the gradient signals are inconsistent and do not accumulate into lasting improvement.

Data Augmentation Fails to Improve Learnability or Gradient Similarity

Section 5.3 tests whether providing semantically similar training data can help the model overcome representation flaws on unlearnable examples. The results are uniformly negative:

Training reward on unlearnable examples stays flat (Figure 8). Under all three augmented data compositions ( $\mathcal{D}_u + \mathcal{D}^{\text{sim}}_u$ , $\mathcal{D}_u + \mathcal{D}^{\text{sub}}_u$ , $\mathcal{D}_u + \mathcal{D}^{\text{sim}}_u + \mathcal{D}^{\text{sub}}_u$ ), the training reward on the original unlearnable examples remains low and essentially unchanged from the no-augmentation baseline. The curves (the three lines in Figure 8, left y-axis) overlap heavily and never rise above approximately 0.1–0.15 average reward.

Validation performance saturates early and overfits (Figure 8, right y-axis). For the combined augmented setting, pass@1 and pass@16 on the validation set are tracked alongside training reward. Both metrics show some initial improvement up to about 20–25 training steps, then plateau and begin declining — the model is overfitting to the training data without genuine generalization. The final validation performance is not substantially better than baseline.

Subproblem learning does not transfer (Figure 9). This is the most informative result. The augmented subproblems ( $\mathcal{D}^{\text{sub}}_u$ , green curve) show the best learnability of any group — their training reward rises smoothly to approximately 0.7 by step 100, exceeding even the original learnable examples (orange curve, which rises to approximately 0.45). The model genuinely learns to solve the subproblems. However, this learning has zero effect on the original unlearnable examples (red curve, which stays flat near 0.1). The augmented similar problems ( $\mathcal{D}^{\text{sim}}_u$ , purple curve) show intermediate learnability (rising to approximately 0.3 by step 100), but again with no transfer to the originals.

Gradient similarity with augmented data does not predict similarity with training distribution for learnable examples (Figure 10). For learnable examples (Figure 10b), the correlation between average gradient similarity to the original training distribution and average similarity to the augmented similar data is essentially zero (R² = 0.090). The augmented data's gradients are aligned with the broader distribution regardless of the specific original example's alignment — the augmented data is "well-behaved."

For unlearnable examples (Figure 10a), the correlation is strong (R² = 0.704). If the original unlearnable example has low similarity to the broader distribution, its augmented similar problems also have low similarity — the gradient outlier property is intrinsic to the specific example and persists across semantically identical variations.

Together, these results demonstrate that data augmentation fails at the representation level, not at the semantic level. The augmented data is high-quality (the model learns it smoothly, Figure 9), but the gradient signals it produces do not transfer to the original unlearnable examples because those examples are fundamentally misaligned in the model's representation space. This is a profound negative result: it means even perfectly constructed similar problems cannot help if the original example has a flawed internal representation.

Mid-Training Improves Gradient Similarity

Section 5.4 provides the positive contrast — an intervention that can improve gradient alignment for difficult examples. Comparing three models:

Figure 11 — Gradient similarity distributions:

Llama-3.2-3B-Base (no mid-training): The distribution of gradient similarity for difficult examples with the broader 500-example set is broad and centered around 0.2–0.3, with a long tail extending below 0.1. Many difficult examples have very low alignment.
OctoThinker-3B-Hybrid-Base (20B tokens mid-training, hybrid data mixture): The distribution shifts rightward, centered around 0.3–0.4 with the lower tail substantially reduced. Overall similarity is consistently higher than the base model.
OctoThinker-3B-Long-Base (20B tokens mid-training, long-context data mixture): Similar improvement over the base model, with the distribution centered around 0.3–0.4 and reduced lower tail.

Both mid-trained models exhibit higher gradient similarity than the base model, indicating that mid-training reshapes representations so that difficult examples become better integrated with the broader training distribution. This is the key evidence that gradient alignment is not a fixed property of model-data pairs but can be improved through appropriate pretraining-stage interventions.

Curriculum Learning Does Not Improve Unlearnability or Reasoning Quality

Appendix A.5 tests whether curriculum learning — training on easier data first, then introducing unlearnable examples — can scaffold learning. The results:

Training reward on unlearnable examples after curriculum (Figure 18a). When unlearnable examples are introduced in Stage 2 (after the model has converged on easy and learnable data), their training reward remains flat — essentially identical to the baseline where unlearnable examples were trained from the beginning. The x-axis shows epochs on the unlearnable examples specifically, confirming they are not ignored but fail to benefit from training.

Reasoning quality after Stage 1 (Figure 18b). After training on easy and learnable examples (Stage 1), reasoning quality on unlearnable examples (which were held out) remains low — concentrated at scores of 0–3, similar to initial quality distributions. The model's improved reasoning on learnable and easy examples does not transfer to the unlearnable ones even indirectly (through shared parameters).

Validation accuracy drop. The paper notes an "obvious drop in validation accuracy when switching to the second stage of training" (Appendix A.5), suggesting that introducing unlearnable examples may actively degrade performance on other data, possibly because their noisy gradients interfere with the representations the model developed during Stage 1.

Ablation Studies and Robustness Checks

Positive-rollout-ratio control (Figure 2): The oversampling-with-replay experiment (Section 4.1) itself serves as an ablation of the standard GRPO setting, testing whether the specific $k_{\text{pos}}=1$ ratio produces different dynamics than the natural distribution. The finding that it slows learnable-group learning without helping the unlearnable group confirms that reward ratio is not the causal factor. The paper also explores two stronger positive-signal interventions — SFT on distilled responses and $k=64$ RL (Appendix A.2) — both of which fail, adding robustness across supervision modalities.
Clipping and KL penalty ablation (Figure 5): Each regularization mechanism is ablated independently (clip-higher, no KL loss), and both simultaneously, showing no differential effect on unlearnable examples. This is a clean ablation because each intervention changes only the component of the loss function hypothesized to cause the problem, leaving the rest of the training pipeline identical.
Reference log-likelihood distribution (Figure 3): This is a diagnostic check that validates the clipping-ablation result — if reference probabilities had been systematically lower for unlearnable examples, the clipping and KL ablations might have been expected to help, but the null result on clipping would have been puzzling. The comparable distributions explain why the ablations don't help: there is no suppression to remove.
Clipping ratio monitoring (Figure 4): Tracking clipping ratios during training provides a dynamic view complementing the static reference-log-likelihood measurement. Even if reference probabilities were comparable at initialization, clipping ratios could diverge during training if the policy's probability of correct rollouts on unlearnable examples evolved differently. The tracking shows this does not occur.
Gradient interference analysis at two scales (Figure 16): By measuring both within-prompt (same example's correct vs. incorrect rollouts) and cross-prompt (all correct vs. all incorrect across the batch) similarity, the paper rules out both local and global gradient interference mechanisms. The finding that unlearnable examples maintain positive within-prompt similarity at step 50 (when learnable examples have shifted to negative) is an unexpected result that reinforces the representation-quality interpretation — the model cannot even distinguish correct from incorrect on these examples.
LoRA vs. full-parameter gradient similarity correlation: The paper verifies that LoRA-based gradient similarity is highly correlated with full-parameter similarity, validating the computational approximation used throughout Section 5. This is a critical robustness check because all gradient-similarity conclusions depend on the LoRA approximation preserving relative similarities across examples. Without this check, it would be possible that LoRA gradients differ systematically from full gradients in ways that create spurious group differences.
Multiple model families and datasets (Table 1): The unlearnability phenomenon is replicated across Qwen (two scales) and Llama model families, and across MATH and DeepScaleR datasets, with consistent qualitative patterns. The Llama-specific results in Appendix A.1 (Figures 12–14) mirror the main-text Qwen results (Figures 2, 3–5, 9–10), showing that gradient similarity distributions, oversampling intervention failure, and data augmentation failure all replicate. This rules out the possibility that unlearnability is specific to Qwen architectures or MATH-level difficulty distributions.
Intersection-based categorization across three runs (Section 3.2): The paper addresses the inherent noise in RL training by requiring consistent classification across three independent runs. An example that is "unlearnable" in one run but "learnable" in another (perhaps due to different random seeds affecting exploration) is excluded from both groups. This means the analyzed subsets represent the stable extremes of the distribution — consistently unlearnable and consistently learnable examples. The tradeoff is that the true fraction of unlearnable examples may be larger than reported (since borderline cases are excluded), but the analyzed groups are cleaner for diagnostic purposes.
GPT-5-mini reasoning quality evaluation with Gemini-2.5-pro validation (Section 5.2, Appendix C): While GPT-5-mini is an imperfect judge, the paper takes steps to validate its assessments indirectly. Augmented data generated by GPT-5 is cross-validated by Gemini-2.5-pro for correctness, ensuring that the quality of augmented data is not itself the confound. For reasoning quality scoring, the relative comparisons (across groups at the same time, and within groups across time) control for systematic biases in GPT-5-mini's scoring — if GPT-5-mini systematically underscores certain types of reasoning, this would affect all groups and time points equally.
Augmented data types (similar problems vs. subproblems, Figure 9): The paper tests two conceptually different augmentation strategies — one that preserves problem structure (similar problems) and one that decomposes it (subproblems). The differential learnability of these augmented types (subproblems are learned much better than similar problems, yet neither transfers) provides evidence that the failure is not about augmentation quality or type — it's about transfer from augmented data to originals, which is blocked regardless of how well the augmented data itself is learned.
Mid-training data mixture variants (Figure 11): The comparison of two OctoThinker variants with different mid-training data mixtures (Hybrid vs. Long) shows that both improve gradient similarity over the base model, suggesting the improvement is robust to the specific mid-training recipe — the key factor is simply doing some mid-training on domain-relevant data.
Curriculum learning stage design (Figure 18): The two-stage design where unlearnable examples are reintroduced after convergence on easy/learnable data tests whether prior learning creates a "scaffold" that enables subsequent learning on unlearnable examples. The negative result (no improvement in reward, drop in validation accuracy) indicates that not only does prior learning not help — it may make things worse, possibly because the representations optimized for easy/learnable data are incompatible with the representations needed for unlearnable examples.
Negative result: ReST $^{EM}$ -style revision model degrades (Appendix K, Figure 16 of the main paper — note: this figure reference is from the paper's appendix and does not appear in the provided text, but is mentioned in the prior sections as an attempted revision-model optimization that backfired): While not directly part of the provided paper content beyond a mention in the prior sections, this serves as an additional robustness check that different RL training paradigms (beyond GRPO) also struggle with unlearnability-related phenomena.

Critical Assessment

The paper makes four central claims, and the experimental evidence for each warrants careful evaluation:

Claim 1: "A substantial subset of training examples remains unlearnable throughout GRPO training despite observing correct rollouts." The evidence for the existence of the unlearnability phenomenon is strong. Table 1 shows consistent percentages across three model families and two datasets, and the intersection criterion across three independent training runs makes the classification robust to training stochasticity. Figures 1b, 2, 5, and 8 all show flat or near-flat training reward curves for the unlearnable group specifically, while learnable examples improve in the same training runs — this within-training-run contrast is the strongest evidence that something genuinely distinguishes these examples rather than the training simply being insufficient. However, two caveats should be noted:

First, the $\tau=0.1$ threshold is arbitrary, and the paper acknowledges this. The exact fraction of "unlearnable" examples depends on this threshold. A lower threshold would produce a smaller unlearnable set; a higher threshold, a larger one. The paper does not report how sensitive the percentages are to threshold choice. However, the gradient similarity and reasoning quality analyses compare distributions of continuous properties across groups, not just group membership — Figure 1c shows that the gradient similarity distribution for unlearnable examples is shifted relative to learnable examples, not just that the means differ. This continuous separation provides robustness: the groups are genuinely different in the underlying properties, not just threshold artifacts.

Second, all experiments use a fixed training budget (the models are trained until validation performance saturates, Section 3.2). It is possible that some currently "unlearnable" examples would become learnable with much longer training — the paper's footnote 1 notes that "when visualizing training rewards for prolonged training steps, some of the unlearnable examples see a sharp increase in reward, accompanied with drop in validation performance." This is ambiguous: it could mean some unlearnable examples are actually learnable with extreme training budgets (just very slowly), or it could mean the sharp increase represents overfitting without genuine generalization. The paper interprets it as the latter (overfitting), but the evidence distinguishing these possibilities is not presented.

Claim 2: "Existing optimization and sampling techniques fail to resolve unlearnability." The evidence for this claim is comprehensive and convincing. The paper tests four categories of optimization-side fixes — reward scarcity (Section 4.1 with three sub-experiments), gradient regularization (Section 4.2 with two ablations plus diagnostic checks), gradient interference (Appendix A.3 with two analyses), and curriculum learning (Appendix A.5) — and all produce negative results. The negative results are clean: the training reward curves for unlearnable examples overlap nearly perfectly with baseline in every intervention (Figures 2, 5, 15b, 18a), while the same interventions affect learnable examples (e.g., oversampling slows them down, clip-higher and no-KL have no effect on them either, curriculum helps them in Stage 1). This provides a natural control — the interventions are "working" in the sense that they change the optimization landscape, but they don't close the unlearnable-learnable gap.

A potential limitation is that all interventions are tested only on Qwen2.5-0.5B (the main text figures; Llama results for some interventions are in Appendix A.1). The paper shows that the oversampling and gradient regularization results replicate for Llama-3.2-3B-Instruct (Figure 12), but the gradient interference and curriculum learning replications are not explicitly reported for Llama. Given the consistency of other Llama replications (Figures 12–14), it is plausible these would also replicate, but it is not demonstrated.

A more fundamental limitation: the paper does not test all possible optimization improvements — it tests the most natural and commonly proposed ones. It is possible that some other optimization technique (e.g., process-based reward models, token-level credit assignment, or architecture modifications) could help. The paper's conclusion is properly scoped: "current" RL approaches show limitations, not "all possible" RL approaches.

Claim 3: "Unlearnable examples have fundamental representation issues characterized by low gradient similarity and ungeneralizable reasoning patterns." The gradient similarity evidence (Figures 1c, 6, 10, 17) is the strongest part of the paper, and the LoRA-based approximation is validated against full-parameter similarity. The causal interpretation — that low gradient similarity causes unlearnability rather than being an effect of it — rests on two observations: (1) similarity is measured at initialization before any RL training, establishing temporal precedence, and (2) mid-training increases both gradient similarity (Figure 11) and downstream RL performance (from Wang et al., 2025), establishing that the property is causally upstream of learnability. These are strong but not definitive causal evidence — an interventional study that directly increases gradient similarity for specific examples and then measures whether those examples become learnable would be stronger, but is likely computationally infeasible at this scale.

The reasoning quality evidence (Figures 1d, 7, Table 2) is more suggestive than definitive. GPT-5-mini scores are noisy, and the paper does not report inter-annotator agreement or validate the scores against human judgments. The concrete examples (Tables 2, 5–7 in the appendix) are compelling illustrations, but the quantitative differences between groups — while visually clear in the box plots — would be stronger if accompanied by statistical tests.

A missing analysis: the paper does not investigate why unlearnable examples have bad representations. Do they share linguistic properties (unusual phrasing, rare vocabulary)? Do they involve reasoning steps that are underrepresented in pretraining? Do they require composition of multiple skills that the pretrained model hasn't learned to combine? Understanding the source of representation flaws would strengthen the diagnostic value and suggest more targeted pretraining interventions. The data augmentation result (Figure 10: semantically similar problems have similar gradient outlier properties) suggests the flaw is not about surface features, but the paper does not dig deeper into what it is about.

Claim 4: "Substantial mid-training improves gradient similarity of difficult examples." The evidence in Figure 11 is limited but suggestive. The comparison involves only three models (one base, two mid-trained variants), all from the same family (Llama-3.2-3B). Both mid-trained models show higher similarity, but the effect size is modest — the distributions shift from centered around 0.2–0.3 to 0.3–0.4, which is a meaningful but not dramatic improvement. The paper does not report whether this improvement translates to reduced unlearnability rates under RL (which would require running full GRPO training on the OctoThinker models and measuring unlearnable fractions — a computationally expensive experiment that is beyond the paper's scope but would have been highly informative). The claim stands as plausible and consistent with the broader narrative, but the causal chain (mid-training → higher gradient similarity → fewer unlearnable examples under RL) is not fully closed.

Overall strengths of the experimental design:

Comprehensive hypothesis elimination: The paper systematically rules out the four most natural optimization-side explanations before pivoting to the representation hypothesis. This methodology — falsifying alternative explanations before advancing a novel one — is rigorous and gives the representation hypothesis stronger standing than if it were presented without ruling out alternatives.
Multi-model, multi-dataset replication: The core phenomena (unlearnability percentages, gradient similarity distributions, intervention failures) replicate across Qwen and Llama families and across MATH and DeepScaleR datasets. This is stronger external validity than most LLM training dynamics papers provide.
Conservative categorization protocol: The intersection-across-runs criterion for defining unlearnable examples removes noise and makes the analyzed groups cleaner, even if it underestimates true unlearnable fractions. This is a methodological strength, not a weakness — it means the effects reported are likely lower bounds on the true phenomenon.
Converging evidence from gradient and behavioral analyses: The gradient similarity findings (Section 5.1) and the reasoning quality findings (Section 5.2) provide independent, convergent evidence for the representation-flaw hypothesis. The gradient analysis shows optimization-space misalignment; the reasoning analysis shows behavioral incoherence. Together, they paint a consistent picture of examples where the model's internal processing is fundamentally flawed.
LoRA approximation validation: The paper validates that LoRA-based gradient similarity correlates with full-parameter similarity, which is essential for the credibility of the gradient analysis given the computational constraints.

Genuine weaknesses and missing experiments:

Small test sets for some analyses: The gradient similarity analysis uses 100 examples per group (300 total). The reasoning quality analysis uses 100 per group. These are reasonable sample sizes but limit the statistical power for fine-grained analyses (e.g., subgroup analyses within the unlearnable group).
No human validation of reasoning quality scores: GPT-5-mini is used as the sole judge of reasoning quality. While relative comparisons control for systematic biases, human validation would strengthen confidence that the quality differences are genuine rather than artifacts of GPT-5-mini's scoring tendencies.
Unaccounted compute for difficulty estimation: The paper does not explicitly account for the cost of computing pass@1 at initialization (2048 samples) or at the end of training (32 samples) when computing unlearnable fractions. This is not a major issue since the paper's contribution is diagnostic rather than a proposed deployment system, but it's worth noting.
Limited mid-training analysis: The mid-training result (Figure 11) involves only two mid-trained variants of a single base model. It would be stronger with more mid-training configurations, ablation of mid-training data quantity and composition, and — ideally — a demonstration that improved gradient similarity from mid-training translates to reduced unlearnability rates under subsequent RL.
No combination of search and revisions with unlearnability analysis: The paper studies GRPO but not PRM-guided search or iterative revision models in combination. This is noted in the prior sections as a limitation of the RLVR literature broadly — the paper's analysis of unlearnability might have different dynamics under different RL paradigms, but this is not explored.
Pass@k degradation not systematically studied: The paper notes that "prolonged training steps, some of the unlearnable examples see a sharp increase in reward, accompanied with drop in validation performance" (footnote 1). This suggests a possible overfitting regime where the model memorizes correct answers on unlearnable examples without generalization. The paper does not systematically analyze this regime — e.g., at what point it occurs, whether it's consistent across examples, or whether it correlates with gradient similarity.
No analysis of whether unlearnable examples share structural properties: The paper shows that unlearnable examples are gradient outliers and have poor reasoning quality, but does not investigate whether these examples share features that predict unlearnability — e.g., specific MATH difficulty levels, problem types (algebra vs. geometry vs. combinatorics), required reasoning depth, or linguistic properties. Such an analysis could make the diagnostic more actionable.

Conditional nature of the claims:

The paper's claims hold most strongly under the following conditions, which are largely the experimental settings used:

RL algorithm is GRPO with dynamic sampling and binary outcome reward. The paper does not test whether unlearnability persists under process-based reward models (PRMs) that provide intermediate feedback, which Sun et al. (2025a) showed can make very hard examples learnable. The extent to which improved reward signals would reduce the unlearnable fraction is unknown.
Training budget is moderate (until validation saturation). The paper does not test whether extreme training budgets (e.g., 10× more steps) would eventually convert some unlearnable examples to learnable — though footnote 1 suggests that when they do improve, it comes with validation degradation (overfitting), not genuine learning.
Models are small-to-medium scale (0.5B–3B). The paper does not test whether unlearnability rates change with model scale. Larger models with better pretrained representations might have fewer gradient outliers, or might have more — the direction is not obvious.
Domain is mathematical reasoning. The paper uses only MATH and DeepScaleR (math datasets). Whether unlearnability exists in other RLVR domains (code generation, agentic tasks, science QA) is not tested, though there is no obvious reason it would be math-specific.

Overall, the experimental evidence strongly supports the paper's diagnostic claims — unlearnability is a real and widespread phenomenon, it cannot be explained by standard optimization-side factors, and it correlates with gradient misalignment and poor reasoning quality. The paper is more tentative about solutions (mid-training improves alignment but the full causal chain to reduced unlearnability is not closed), which is appropriate given the scope of the study. The experiments are carefully designed, the negative results are clean and informative, and the convergence of gradient-based and behavioral evidence makes the representation-flaw interpretation compelling. The primary limitation is that the paper diagnoses the problem without fully characterizing its scope (what fraction of examples are curable by mid-training, how this depends on model scale) or providing a complete solution — but this is characteristic of foundational diagnostic work that opens a new research direction rather than closing one.

6. Limitations and Trade-offs

Fundamental Assumption: Difficulty Can Be Estimated Cheaply Enough to Be Practical

The assumption or constraint. The entire unlearnability framework — from the working definition (Section 3.2) to the final analysis — relies on identifying which examples are unlearnable through pass@1 estimation that requires sampling 32 responses per example at convergence and 2048 responses per example at initialization for the baseline pass-rate estimation. While the paper does not propose a deployment methodology, its diagnostic value depends on being able to measure unlearnability without incurring prohibitive computational cost. The paper does not address this cost explicitly, but the operational definition in Section 3.2 makes clear that identifying unlearnable examples requires training a model to convergence (to measure final pass@1), then sampling 32 responses per candidate example to classify it. For the 500-question MATH Easy test set with 1024-sample difficulty estimation per question, this is a substantial overhead above training itself.

The consequence. The diagnostic framework is expensive to apply in practice. A practitioner running GRPO training on a new dataset would need to (a) complete full RL training to convergence, (b) sample 32 responses per example across the entire training set to estimate final pass@1, and (c) run three independent training runs to get the intersection-based stable classification (Section 3.2). The cost of (c) alone triples the training compute budget. This means the unlearnability diagnostic is primarily a retrospective analysis tool for understanding completed training runs — it cannot be used cheaply during training to dynamically filter or reprioritize examples. The paper's gradient similarity diagnostic (Section 5.1) partially addresses this by offering a pre-RL predictor, but computing gradient similarity at scale (100 examples × 1000 rollouts × LoRA gradients) is itself computationally intensive.

What evidence exists in the paper. The paper explicitly notes in Section 3.2 that the pass@1 threshold $\tau=0.1$ is "rather arbitrary" and that "the intersection of the different groups of examples as the final subject for analysis to reduce noise" requires three independent training runs. Table 1's percentages are based on this triple-run intersection, which is the most rigorous classification but also the most expensive. The paper does not report wall-clock time, GPU-hours, or total generation cost for any diagnostic step. The gradient similarity analysis (Section 5.1) is described as using "a fixed, randomly initialized LoRA adapter" for "computational efficiency," but no absolute cost figures are given — the LoRA approximation makes the computation feasible, not necessarily cheap, since 1000 rollouts per example is still substantial.

Mitigation status. The paper does not address this limitation directly. Section 8 (Discussion) mentions that "identifying which rollout features beyond outcome correctness contribute to smooth and generalizable LLM optimization represents a promising direction for future research," which could lead to cheaper diagnostics, but no concrete approach is proposed. The gradient similarity analysis (Section 5.1) offers a route to pre-RL prediction that is cheaper than full training but still expensive. The paper provides no lightweight proxy for unlearnability (e.g., a classifier trained on question text, or a quick version of the gradient similarity metric using fewer rollouts).

Scope Limited to Small-to-Medium Scale Models on Mathematical Reasoning Only

The assumption or constraint. All experiments use models in the 0.5B–3B parameter range (Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-3B-Instruct) on two mathematical reasoning datasets (MATH and DeepScaleR). The paper does not test whether the unlearnability phenomenon persists, weakens, or strengthens at larger model scales (7B, 13B, 70B+) or in other reasoning domains (code generation, logical reasoning, scientific QA, agentic tasks). The paper acknowledges this constraint in Section 6: "Due to resource limit, our experiments are conducted on small to mid-scale base and instruct models in the mathematical reasoning domain."

The consequence. Three important external validity questions are left unanswered. First, scale dependence: larger models typically have better pretrained representations and may process "hard" examples in ways that produce more aligned gradients. If unlearnability diminishes with scale — e.g., most examples that are unlearnable at 0.5B become learnable at 7B — then the phenomenon is primarily relevant for small-model regimes and less concerning for state-of-the-art training pipelines using larger models. Conversely, if unlearnability persists or even intensifies at scale (because larger models have more capacity to develop idiosyncratic representations), then the finding is more broadly significant. The paper provides no evidence either way. Second, domain dependence: mathematical reasoning has clean binary outcome verification, well-defined reasoning steps, and a particular structure (symbolic manipulation, multi-step deduction). Other RLVR domains — code generation (where partial correctness and execution-based rewards provide richer signals), agentic tasks (where trajectories are longer and rewards sparser), or scientific QA (where reasoning patterns differ) — might exhibit different unlearnability dynamics. Third, base vs. instruct models: the paper uses Qwen2.5-0.5B (base), Qwen2.5-3B (base), and Llama-3.2-3B-Instruct, but does not systematically compare base vs. instruct variants of the same model, leaving open whether instruction tuning affects unlearnability rates.

What evidence exists in the paper. The multi-model replication (Table 1) shows that unlearnability exists across both Qwen and Llama families, which provides some cross-architecture evidence. However, these are all 0.5B–3B models, and the Llama-3B-Instruct result (21.9% unlearnable) is for a different difficulty distribution (MATH Hard, levels 3–5) than the Qwen-0.5B result (30.2% unlearnable on MATH Easy, levels 1–4), making direct comparisons difficult. The DeepScaleR result (Qwen2.5-3B, 16.7% unlearnable) uses a different dataset entirely. The mid-training analysis (Section 5.4, Figure 11) involves Llama-3.2-3B and its mid-trained variants, so even that analysis stays within the 3B scale. The paper does not cite or conduct experiments at larger scales, nor does it discuss expected scaling behavior.

Mitigation status. The paper acknowledges this limitation explicitly in Section 6, listing it as the first limitation: "Due to resource limit, our experiments are conducted on small to mid-scale base and instruct models in the mathematical reasoning domain." No specific future work is proposed to test scale or domain generalization — the acknowledgment is a disclaimer, not a research agenda. The paper's reference to "mid-scale" models is accurate (0.5B–3B is indeed small-to-mid by current standards), but this means the findings should be treated as established for this regime and plausible but unverified for larger-scale or different-domain settings.

The Operational Definition of Unlearnability Is Threshold-Dependent and May Miscategorize Examples That Would Learn With More Training

The assumption or constraint. The working definition in Section 3.2 classifies an example as unlearnable if its final pass@1 is below $\tau=0.1$ (estimated with $N=32$ samples) after training to convergence "when validation performance saturates." Both the threshold and the convergence criterion are operational choices that affect which examples are labeled unlearnable. The paper explicitly states: "Our working definition of unlearnability relies on a thresholded notion of convergence and is therefore a useful operational tool rather than a sharp categorical distinction, and a fraction of examples will sit near the boundary under any such criterion" (Section 6). Additionally, footnote 1 notes: "When visualizing training rewards for prolonged training steps, some of the unlearnable examples see a sharp increase in reward, accompanied with drop in validation performance."

The consequence. Two related issues arise. First, sensitivity to training budget: the paper trains until validation performance saturates, but "saturation" is not defined quantitatively — there is no early-stopping criterion, no patience parameter, and no specification of how many steps constitute "prolonged" training. If some examples classified as unlearnable would eventually become learnable with significantly more training (e.g., 5× the current budget), then the unlearnable category conflates "slow to learn" with "fundamentally unlearnable." The footnote's observation about sharp reward increases during prolonged training could be interpreted either as delayed genuine learning or as overfitting — the paper interprets it as overfitting because it is "accompanied with drop in validation performance," but the evidence distinguishing these interpretations is not presented. Second, boundary instability: examples near the $\tau=0.1$ boundary (e.g., final pass@1 of 0.09 vs. 0.11) may be classified differently due to the 32-sample estimation noise rather than genuine differences in learnability. The intersection criterion across three runs mitigates this somewhat, but an example that achieves pass@1 of 0.12 in two runs and 0.08 in one run would be excluded from both unlearnable and learnable groups due to inconsistency — the paper does not report how many examples fall into this excluded middle category.

What evidence exists in the paper. The paper is transparent about the threshold's arbitrariness (Section 3.2: "Since the pass@1 threshold is chosen rather arbitrarily...") and the operational nature of the definition (Section 6: "a useful operational tool rather than a sharp categorical distinction"). The intersection criterion across three runs is described in Section 3.2: "we conduct three independent GRPO trainings for each setting and take the intersection of the different groups." However, the paper does not report sensitivity analysis — e.g., how unlearnable fractions change with different $\tau$ values, different $N$ values for pass@1 estimation, or different training budgets. The footnote about prolonged training is the only glimpse into what happens beyond the convergence horizon, and it raises more questions than it answers. The continuous analyses (gradient similarity distributions, reasoning quality scores) provide robustness by showing distributional separation between groups rather than relying solely on the binary classification, but the headline percentages in Table 1 remain threshold-dependent.

Mitigation status. The paper acknowledges this limitation explicitly both in Section 3.2 ("chosen rather arbitrarily") and Section 6 ("not a sharp categorical distinction, and a fraction of examples will sit near the boundary"). The gradient similarity and reasoning quality analyses (Sections 5.1–5.2) partially mitigate the concern by demonstrating that unlearnable and learnable groups differ on continuous properties, not just the thresholded pass@1 — this suggests the binary classification captures genuine underlying differences, even if the boundary is fuzzy. However, the paper does not provide a sensitivity analysis, does not propose a more principled threshold selection method, and does not distinguish "slow to learn within a realistic budget" from "impossible to learn under any budget" — the distinction matters practically because extremely slow learners might still be worth including if sufficient compute is available.

The Gradient Similarity Diagnostic Is Correlational, and the Causal Argument — That Low Similarity Causes Unlearnability — Is Not Fully Closed

The assumption or constraint. The paper's central diagnostic claim is that "unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples" (Abstract). The evidence for this claim is that gradient similarity measured under the initial policy correlates with later unlearnability (Figures 1c, 6). The causal interpretation — that low gradient similarity causes unlearnability, and that fixing gradient similarity would fix unlearnability — relies on temporal precedence (similarity is measured before RL) and the mid-training result (Figure 11: mid-training increases similarity and improves RL outcomes, per Wang et al., 2025). However, these are two points of evidence, not a direct causal demonstration.

The consequence. Without a direct interventional study — experimentally manipulating gradient similarity for specific examples and measuring whether those examples become learnable — the causal direction could run differently. Alternative explanations compatible with all observations include: (1) a third factor (e.g., intrinsic difficulty, required reasoning depth, or pretraining data frequency) causes both low gradient similarity and unlearnability independently, so increasing similarity without addressing the root cause might not help; (2) gradient similarity is a symptom of deeper representation properties that RL cannot fix, and the same properties would prevent learning even if similarity were artificially increased; (3) the correlation is strong for the extremes but breaks down for intermediate examples, making similarity an imperfect predictor. The mid-training result (Figure 11) is suggestive but limited — it shows that mid-training increases similarity in aggregate and that mid-trained models perform better under RL (per Wang et al., 2025), but it does not show a per-example relationship between similarity improvement and learnability improvement.

What evidence exists in the paper. The paper provides strong correlational evidence: the separation in gradient similarity distributions between unlearnable and learnable examples (Figure 1c, Figure 6), the persistence of low similarity through training (Figure 17), and the failure of data augmentation to improve either similarity or learnability (Section 5.3, Figures 8–10). The mid-training experiment (Figure 11) is the closest the paper comes to a causal demonstration, but it shows only aggregate similarity improvement in mid-trained models — it does not track individual examples from low-similarity at the base stage to high-similarity after mid-training and then demonstrate those specific examples become learnable. The paper does not report per-example similarity changes from mid-training or the correlation between similarity improvement and RL performance improvement at the example level.

Mitigation status. The paper does not claim to have demonstrated causality definitively, and its language is appropriately modulated: the Abstract says unlearnable examples "have fundamental representation issues, characterized by low gradient similarity" (correlational language), and Section 5.4 says "Mid-training can increase gradient similarity" (observational, not claiming the increase causes learnability). The Discussion (Section 6) frames the finding as revealing a "failure mode" — "When the underlying representation is flawed to begin with, outcome-based RL has no clear path to repair it" — which is a mechanistic hypothesis, not a proven causal law. The paper is stronger on diagnosis than on proof of causation, which is appropriate for an initial characterization study but leaves the causal mechanism as an open question for future work with more controlled interventions.

Data Augmentation's Failure Defines a Boundary but Does Not Explain It — The Representation Flaw Is Diagnosed but Not Characterized

The assumption or constraint. The paper demonstrates that data augmentation (similar problems and subproblems) fails to improve learnability or gradient similarity for unlearnable examples (Section 5.3), and that semantically similar problems can have very different gradient signatures (Figure 10). This establishes that the representation flaw is not about problem structure or surface features — it is about something deeper in how the pretrained model processes specific examples. However, the paper does not characterize what this deeper property is. The paper does not analyze whether unlearnable examples share linguistic features (unusual vocabulary, syntactic complexity, ambiguous phrasing), problem-type features (algebra vs. geometry vs. combinatorics), required reasoning depth (number of steps, number of subproblems), or pretraining data statistics (frequency of similar problems in the pretraining corpus).

The consequence. The diagnostic identifies that unlearnable examples exist and that their representations are flawed, but provides no actionable guidance for predicting unlearnability before RL training (beyond expensive gradient similarity computation) or for preventing it through targeted pretraining data curation. A practitioner who knows that 16–30% of their difficult examples are unlearnable still does not know: (a) which examples those will be without running RL training, (b) what properties make those examples susceptible to representation flaws, or (c) what kinds of pretraining or mid-training data would be most effective at improving their representations. The paper's recommendation to invest in mid-training (Section 6) is sound at a high level but unspecific — it does not say what data to use for mid-training, how much mid-training is needed, or how to verify that mid-training has improved representations for the specific examples of concern.

What evidence exists in the paper. The data augmentation results (Figures 8–10) provide strong negative evidence that surface-level similarity does not guarantee gradient similarity for unlearnable examples. The strong R²=0.704 correlation in Figure 10a for unlearnable examples (vs. R²=0.090 for learnable in Figure 10b) shows that gradient outlier status is highly conserved across structurally identical variations for unlearnable examples but not for learnable ones. Table 2 and Appendix D provide concrete examples of flawed reasoning traces, which are illustrative but are selected case studies rather than systematic analysis. The paper does not report any analysis of example-level features (difficulty level, problem type, token length, vocabulary statistics, or pretraining frequency) correlated with unlearnability. The mid-training result (Figure 11) shows that some mid-training improves similarity, but the two OctoThinker variants use different data mixtures and both improve — leaving unclear which aspects of mid-training are responsible.

Mitigation status. The paper acknowledges this gap implicitly in Section 6: "several key questions remain open: what data are most effective for mid-training, and which algorithms best facilitate mid-training." This is an agenda for future work, not a claim to have solved the problem. The paper's contribution is establishing that representation flaws exist and that they resist RL-stage fixes — characterizing the nature of those flaws and designing targeted interventions is left to subsequent research. This is a legitimate scope limitation for a diagnostic paper, but it means that the findings are more useful for understanding current limitations than for prescribing specific solutions. A practitioner who wants to reduce unlearnability in their pipeline based on this paper's findings would know to look at mid-training but would have to experiment largely from scratch to determine what mid-training data and procedure works for their specific setting.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the conversation around RLVR for LLM reasoning from "how do we improve the RL algorithm?" to "what makes an example learnable under RL in the first place?" This is not an incremental refinement — it is a diagnostic paradigm shift that redirects attention from optimization-side tweaks (clipping schedules, exploration bonuses, credit assignment granularity) toward representation quality as the primary bottleneck. Before this work, the field's implicit model was that any example with pass@k > 0 was learnable given sufficient positive reward signal, and that the limiting factor was exploration — getting the model to stumble into correct answers frequently enough. The unlearnability phenomenon breaks that model. Table 1 shows that 16–30% of difficult examples (which do receive positive reward signals) remain flatlined throughout GRPO training across three model families and two datasets. Section 4 systematically eliminates the optimization-side explanations that the field would have reached for first — positive rollout scarcity (Figure 2, Figure 15), gradient regularization (Figures 3–5), and gradient interference (Figure 16) — with clean negative results that converge on a single conclusion: the bottleneck is upstream of RL entirely.

The paper resolves a latent tension in the RLVR literature between optimistic findings (Sun et al., 2025a: "extremely difficult examples can be learned with fine-grained rewards") and pessimistic ones (Yue et al., 2025: "pass@k degrades after RL"; Wu et al., 2026: "RL cannot escape its origin distribution"). These are not contradictory — they describe different regions of a representation-quality spectrum. Sun et al.'s findings apply to examples where representations are good enough that richer reward signals (beyond binary outcome) can provide sufficient learning guidance. Yue et al. and Wu et al.'s findings describe the regime where RL is fundamentally limited to sharpening existing capabilities. This paper adds the missing piece: there exists an intermediate regime — examples with pass@k > 0 but badly misaligned representations — where outcome-based RL provides positive signals that cannot be integrated into coherent learning because the gradient directions they produce are isolated from the broader training distribution (gradient similarity of 0.46 ± 0.2 for unlearnable examples vs. 0.67 ± 0.1 for learnable, Figure 6). The field can now situate prior findings within a unified representation-quality framework rather than treating them as conflicting claims.

This work also changes the perceived division of labor between training stages. The standard pipeline — pretrain, optionally mid-train, then RL — implicitly treats RL as the stage where reasoning capabilities are acquired. The paper's central finding is that RL cannot acquire capabilities for examples with flawed representations; it can only amplify capabilities for examples with already-aligned representations. Section 5.4 (Figure 11) shows that mid-training — a pretraining-stage intervention — can improve gradient alignment, while Sections 4 and 5.3 show that no RL-stage intervention can. This inverts the pipeline's logic: the heavy lifting of ensuring learnability must happen before RL begins, and RL's role is consolidation and amplification of what is already learnable. The practical consequence is that investment should shift toward mid-training data curation, quality filtering based on gradient-alignment diagnostics, and pretraining data mixture optimization — areas previously considered preliminary to the "real work" of RL.

The paper's gradient similarity diagnostic (Section 5.1) provides a cheap pre-RL predictor of learnability that did not previously exist. Computing gradient similarity under the initial policy requires only the pretrained model and training data — no expensive RL training runs — yet it predicts which examples will resist learning (Figures 1c, 6). This transforms data curation for RLVR from a post-hoc "train and see what works" process to a proactive "measure alignment and filter or remediate" process. The finding that semantically similar problems can have very different gradient signatures for unlearnable examples (R² = 0.704 correlation between original and augmented gradient similarity for unlearnable examples vs. R² = 0.090 for learnable, Figure 10) means that data quality cannot be assessed by semantic inspection alone — gradient-based diagnostics capture representation properties that are invisible to human readers but determinative of RL outcomes.

Research directions this work makes more attractive:

Pretraining and mid-training data engineering targeted at representation alignment, rather than continued algorithmic refinement of RL objectives.
Gradient-based data selection and difficulty estimation as preprocessing steps before RL.
Studying what pretraining data properties (frequency, diversity, format) produce well-aligned representations for complex reasoning tasks.
Understanding the relationship between reasoning trace quality and gradient alignment — connecting behavioral properties to optimization geometry.

Research directions this work makes less attractive:

Incremental improvements to GRPO variants (alternative clipping schedules, entropy bonuses, dynamic sampling ratios) — Sections 4.1, 4.2, and Appendix A.3 show these do not address the fundamental bottleneck.
Pushing RLVR to harder problems without first improving representations — unlearnable examples (difficulty bin 5 equivalent) show near-zero improvement regardless of budget, mirroring the paper in the reference example.
Treating all difficult examples as homogeneous — the learnable/unlearnable distinction is critical, and uniform treatment wastes compute on examples that cannot benefit.

Follow-Up Research This Work Enables

Scaling laws for unlearnability: does the phenomenon diminish with model size or training data? The paper's experiments use 0.5B–3B models. A natural extension would replicate the unlearnability analysis across model scales — e.g., Qwen2.5 at 0.5B, 1.5B, 3B, 7B, 14B, all trained on the same MATH subset with identical GRPO hyperparameters. The key measurement would be the fraction of unlearnable examples as a function of parameter count and pretraining FLOPs. If unlearnability is primarily a small-model phenomenon (diminishing sharply with scale because larger models have better-aligned pretrained representations), then the finding is important for efficient deployment but less concerning for frontier training. If it persists at scale, it represents a fundamental ceiling on RLVR effectiveness independent of compute budget. The gradient similarity metric provides the cheap pre-RL diagnostic needed to do this analysis without expensive RL training at each scale — compute similarity at initialization across model sizes and measure whether the fraction of low-similarity examples decreases. A strong follow-up would also vary pretraining data quantity (compute-optimal vs. overtrained models of the same parameter count) to disentangle scale effects from data effects.

Causal intervention: can targeted mid-training on specific unlearnable examples convert them to learnable? Section 5.4 shows that mid-training improves aggregate gradient similarity and that mid-trained models perform better under RL (per Wang et al., 2025), but it does not demonstrate per-example conversion. A strong causal test would identify a set of unlearnable examples in a base model, construct a mid-training corpus specifically enriched with those examples and their close variants, mid-train the base model on this targeted data, recompute gradient similarity for those examples, and then run GRPO to measure whether they become learnable. The key evidence would be a per-example comparison: does the subset of initially unlearnable examples whose gradient similarity increased during mid-training subsequently show improved training reward under RL, while those whose similarity did not change remain unlearnable? This would directly test the causal claim that gradient similarity mediates learnability. The experiment would also reveal how much mid-training data per problematic example is needed to shift representations — data that would directly inform practical pipeline design.

What structural or semantic properties predict gradient outlier status, beyond gradient similarity itself? The paper shows that gradient similarity predicts unlearnability (Figure 1c) and that semantically similar problems can have very different gradient signatures for unlearnable examples (Figure 10a), but does not analyze what features of the problem text or required reasoning correlate with low gradient similarity. A natural follow-up would extract features from unlearnable vs. learnable MATH problems: difficulty level (the dataset's own 1–5 labels), problem type (algebra, geometry, number theory, combinatorics, precalculus), required reasoning depth (number of distinct steps in a reference solution, measured by parsing GPT-5-generated ground-truth solutions), vocabulary rarity (token frequency in the pretraining corpus), syntactic complexity (parse tree depth), and presence of specific mathematical constructs (modular arithmetic, case analysis, geometric visualization). The analysis would answer: are unlearnable examples concentrated in specific MATH categories or difficulty levels? Do they require composition of more subskills? Are they phrased in ways that are distributionally unusual for the pretraining data? The paper's example in Table 2 involves case analysis and spatial reasoning — are these overrepresented among unlearnable examples? This would make the diagnostic more actionable by enabling filtering based on problem features without gradient computation.

Do process-based reward models (PRMs) reduce or eliminate unlearnability? The paper uses binary outcome reward throughout. Sun et al. (2025a) showed that fine-grained reward assignment (rewarding partial progress on intermediate steps) enables learning on extremely difficult examples with near-zero initial pass@k. A direct extension would test whether PRM-guided RL (e.g., using beam search against a process reward model as in the reference example paper) reduces the unlearnable fraction compared to outcome-only GRPO. The hypothesis: PRMs provide credit assignment at the step level, which could give useful gradient signal even when the full solution's gradient is misaligned — individual correct reasoning steps might have better-aligned gradients than the complete solution. The experiment would replicate the paper's GRPO pipeline but with a learned PRM providing token-level or step-level rewards instead of rollout-level binary rewards, then measure unlearnable fractions using the same intersection-based criterion. The gradient similarity analysis could also be recomputed using PRM-based loss gradients to see if step-level gradients are better aligned than rollout-level gradients for unlearnable examples. If PRMs substantially reduce unlearnability, it suggests the problem is partially about credit assignment granularity rather than purely about representation quality. If PRMs don't help, it strengthens the representation-flaw interpretation.

Cross-domain replication: does unlearnability exist in code generation, scientific reasoning, or agentic tasks? The paper's findings are entirely within mathematical reasoning (MATH, DeepScaleR). The phenomenon should be tested in other RLVR domains with binary outcome rewards: code generation (e.g., APPS, MBPP, SWE-bench with execution-based verification), scientific QA (e.g., GPQA with multiple-choice answers), and tool-use/agentic tasks (e.g., WebArena with task-completion rewards). The key measurement is the same: categorize examples by initial success rate, run GRPO to convergence, measure final pass@1, and compute unlearnable fractions (Table 1 equivalent for each domain). Code generation is particularly interesting because execution feedback provides richer signals than binary answer matching — does this make unlearnability less prevalent? For agentic tasks, trajectories are much longer and rewards sparser, potentially making unlearnability more prevalent because the chance of accidentally producing a correct trajectory is lower. Gradient similarity analysis across domains would also reveal whether gradient alignment is a general property or domain-specific — do the same examples have low similarity across all tasks, or is alignment task-dependent?

Can gradient similarity be used as an online filter during RL training to dynamically deprioritize unlearnable examples? The paper's gradient similarity diagnostic requires computing gradients under the initial policy — a pre-RL measurement. A more ambitious extension would develop an online version that estimates gradient alignment during training without requiring separate gradient computation. One approach: track the variance of per-example training rewards over a sliding window of recent steps. The paper observes (Figure 1b) that unlearnable examples have flat reward trajectories while learnable examples show increasing reward. Could reward trajectory flatness serve as a cheap proxy for gradient misalignment that can be computed on-the-fly? The experiment would compare offline gradient similarity (the paper's method) against online metrics (reward variance, reward trend slope, entropy of the policy's output distribution on that example) to see whether any cheap online metric correlates strongly enough with gradient similarity to serve as a real-time filter. If such a proxy exists, it could enable dynamic sampling that reduces the fraction of training compute spent on unlearnable examples without requiring expensive offline preprocessing.

Practical Applications and Downstream Use Cases

Data curation and filtering for RLVR training pipelines. The most immediate practical application is using gradient similarity or a cheaper proxy to filter training data before RL begins. The paper shows that 16–30% of difficult examples are unlearnable (Table 1) and that oversampling them provides no benefit (Section 4.1, Figure 2). Removing these examples from the RL training set before training starts would eliminate wasted compute — if 30% of the difficult subset is unlearnable, filtering them reduces the effective training set size for the difficult portion by nearly a third while maintaining validation performance (since these examples never contributed to validation improvements anyway). For a training run using 1024 prompts per batch, 8 rollouts per prompt, over 100+ steps, this represents a substantial GPU-hour savings with zero accuracy cost. The filtering could be done by computing gradient similarity for all difficult training examples under the initial policy (using the LoRA approximation for efficiency) and removing those below a threshold (e.g., similarity below 0.4, corresponding to the unlearnable cluster in Figure 1c). The paper's finding that gradient similarity measured before RL predicts RL outcomes makes this a pre-training filter rather than a post-hoc analysis.

Mid-training data design targeted at representation alignment. The paper shows that mid-training on 20B tokens improves gradient similarity for difficult examples (Section 5.4, Figure 11) and that mid-trained models benefit more from subsequent RL (per Wang et al., 2025). This provides a concrete objective for mid-training data engineering: rather than optimizing mid-training data for perplexity or downstream accuracy, optimize it for gradient alignment on the target RL training set. A practical pipeline: (1) identify difficult examples in the target RL training data, (2) compute their gradient similarity to the broader dataset under the current base model, (3) construct mid-training data specifically designed to increase similarity for low-similarity examples — e.g., by including many variants of those examples, by including subproblems that decompose their reasoning, or by including examples that require similar reasoning patterns but have simpler surface forms, (4) mid-train, recompute similarity, and verify improvement before proceeding to expensive RL training. This turns mid-training from a generic "warmup" stage into a targeted representation-repair stage. The paper's finding that data augmentation at the RL stage fails (Section 5.3) but mid-training succeeds suggests that the timing of the intervention (before vs. during RL) is critical — representations are plastic during continued pretraining but become frozen during RL optimization.

Compute allocation across training stages informed by unlearnability rates. The paper's finding that unlearnability is fundamentally a representation problem that RL cannot fix suggests a decision rule for allocating a fixed total compute budget across pretraining, mid-training, and RL. If unlearnable fractions are high (e.g., 30%+ of difficult examples), additional RL compute will not help those examples — the marginal return on RL FLOPs for the unlearnable subset is near zero (flat reward curves in Figures 1b, 2, 5, 8). The efficient allocation is to shift compute from RL to mid-training until unlearnable fractions drop to an acceptable level, then apply RL to the now-learnable examples. The paper does not provide the quantitative relationship between mid-training compute and unlearnability reduction, but it provides the diagnostic (gradient similarity) needed to measure it. A practitioner could run a small-scale experiment: measure unlearnable fraction under current mid-training, increase mid-training budget by 2×/4× and remeasure, and determine the elbow point where additional mid-training stops reducing the unlearnable fraction. This prevents the common failure mode of pouring RL compute into examples that gradient analysis could have predicted would never benefit.

Quality-based filtering of correct rollouts during RL training. The reasoning quality analysis (Section 5.2, Figures 1d, 7) shows that unlearnable examples produce correct answers through incoherent, ungeneralizable reasoning — the model "hacks" the binary reward with unreliable heuristics. A practical improvement to the GRPO pipeline would add a reasoning quality check on correct rollouts before using them for gradient updates. Rather than accepting any rollout with correct final answer as a positive example, filter correct rollouts through a lightweight quality scorer (e.g., a fine-tuned classifier trained on the GPT-5-mini annotations from this paper, or a process reward model that checks intermediate step coherence). Rollouts with correct answers but low reasoning quality scores would be treated as neutral (excluded from gradient computation) or negative (treated as incorrect, since the reasoning is unreliable). This would prevent the model from reinforcing the "bag of heuristics" patterns that produce lucky correct answers without genuine understanding. The paper's finding that reasoning quality on learnable examples improves during training (Figure 7, shift from intermediate to high scores) while unlearnable examples stagnate suggests that quality-filtered training would naturally concentrate gradient updates on examples with genuine reasoning, which are also the examples with high gradient similarity. The cost would be the quality scoring step, which could be amortized using a small, fast classifier.