Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models

ArXiv: 2602.12036

🎯 Pitch

Composing existing math problems into chained multi-step prompts—without any new ground-truth data—boosts reinforcement learning performance substantially, with gains scaling from +3.3% to +10.5% pass@1 as models grow from 4B to 30B. A curriculum that progressively increases the number of composed problems lets a mere 4B model outscore several 8B baselines on AIME24, and cross-domain compositions (e.g., physics + math) beat naive domain mixing by over 7%.

1. Executive Summary

This paper introduces Composition-RL, a method that automatically composes multiple existing training prompts into new, harder verifiable questions—for instance, chaining the answer of one math problem as an intermediate variable in a second problem—and uses these compositional prompts for reinforcement learning with verifiable rewards (RLVR) on models from 4B to 30B parameters (Qwen3 series) trained on the MATH12K dataset. Composition-RL consistently improves reasoning over standard RL on the original prompts, with pass@1 gains scaling from +3.3% (4B) to +10.5% (30B-A3B) across math and multi-task benchmarks, and a curriculum variant that progressively increases compositional depth ultimately enables a 4B model to outperform several 8B baselines on AIME24 (37.9% vs. 24.6–34.6%). The approach also extends to cross-domain training, where composing physics and math prompts (Physics-MATH-Composition-141K) outperforms naive mixed or sequential training by +7.1% on AIME24 over the strongest baseline, establishing that prompt composition provides both implicit process supervision and compositional generalization—benefits that compound with model scale and compositional depth.

2. Context and Motivation

The Core Problem: RLVR Training Data Is Finite, and Progressively More of It Goes Stale

The fundamental problem this paper addresses is simple to state but has resisted simple solutions: during reinforcement learning with verifiable rewards (RLVR), a large fraction of training prompts become uninformative as training progresses, and the pool of useful prompts shrinks to a fraction of its original size. This is not a theoretical curiosity—it is an empirical bottleneck that caps the performance of large reasoning models (LRMs) and wastes the expensive computation invested in RL training.

To understand why this happens, consider how RLVR works in practice. At each training step, the model samples a group of responses (typically 8) for each prompt, scores them with a verifier (a rule-based function that checks whether the final answer matches the ground truth), and computes advantage signals from the distribution of scores using GRPO (Group Relative Policy Optimization, Equations 2–3). When all sampled responses for a prompt are either correct (solve_all) or all incorrect (solve_none), the advantage becomes zero for every response—because the advantage is the z-score normalization within the group—and the policy gradient update vanishes entirely (Equation 1). This is what the authors call a "zero-variance" or "zero-advantage" prompt.

The dynamic sampling trick (Equation 5) simply filters these prompts out of the training batch, which prevents wasted computation but does nothing to expand the effective dataset. As the model becomes stronger over the course of training, the solve_all rate rises dramatically. The paper's meta-experiment in Figure 2 (Left) shows the scale: for Qwen3-4B-Base trained on MATH12K, the solve_all ratio climbs from near zero to over 50% within the first 50 steps and stabilizes around 75% thereafter. This means that at later training stages, only about $12{,}000 \times (1 - 0.75) \approx 3{,}000$ prompts remain informative out of the original 12K—a 75% reduction in effective training data. Meanwhile, the solve_none rate remains low (around 5%) at 250 steps, meaning the dominant source of data loss is not hard prompts but easy prompts that the model has already mastered.

This is the gap the paper identifies: the field has focused heavily on solve_none (hard) prompts, but the escalating solve_all rate on easy prompts is arguably the larger and more persistent drain on effective training data.

Why This Problem Matters

The stakes are high for both practical and theoretical reasons.

Practical impact: training data is scarce and expensive. Verifiable prompts are not free. Each requires a question with a ground-truth answer that can be automatically checked—which, for math, means well-formatted competition problems; for science, problems with exact numeric or symbolic answers that rule-based verifiers can handle. The MATH dataset (Hendrycks et al., 2021) contains only 12K training problems after excluding the MATH500 test set. Collecting and cleaning additional high-quality verifiable prompts is costly: the authors cite DeepMath-103K (He et al., 2025b), which required substantial curation effort, and RLV-E (Zeng et al., 2025), which built adaptive verifiable environments. If 3/4 of your carefully curated dataset becomes useless 50 steps into training, that represents enormous wasted effort—and it means models stop learning long before their capabilities would otherwise plateau.

Theoretical significance: it exposes an implicit curriculum collapse. During standard RLVR, the model naturally experiences a form of automatic curriculum: early in training, many prompts are at the right difficulty level (neither trivially easy nor impossibly hard), providing rich learning signals. As the model improves, the curriculum collapses—easy prompts become solve_all, hard prompts remain solve_none, and the effective training distribution becomes increasingly narrow. This is not a flaw in any particular RL algorithm; it is a fundamental consequence of training on a fixed prompt set with a binary verifier. The paper's contribution can be understood as manually reconstructing an appropriate curriculum by composing prompts to keep them at the frontier of the model's capability.

Connection to scaling. Prior work on scaling laws for RLVR has shown that performance improves with longer training (Liu et al., 2025b; He et al., 2025a), but this work implicitly relies on having enough informative prompts to sustain training. If the effective dataset size collapses, scaling training steps alone cannot help—the model simply sees the same small set of informative prompts repeatedly. Composition-RL offers a complementary axis: it scales the effective data complexity in tandem with the model's capability, enabling continued improvement without requiring new data collection.

Where Prior Approaches Fall Short

The paper identifies three broad categories of prior work, each of which addresses part of the problem but leaves the solve_all issue unresolved.

1. Better RL algorithms for zero-variance prompts. Several recent works have proposed modifications to the advantage computation that produce non-zero signals even when all group members have the same verifier score. For example, NGRPO (Nan et al., 2025) uses negative reinforcement signals for solve_none prompts; RL-ZVP (Le et al., 2025) shapes advantages based on response entropy; Beyond-80/20 (Wang et al., 2025) focuses on high-entropy minority tokens that drive effective learning. These methods improve training stability and can extract useful gradients from solve_none prompts, but they are fundamentally designed for the hard-prompt problem. They do nothing to address the easy-prompt problem, because the advantage signal from solve_all prompts remains zero regardless of how you reshape it—there is no variation to exploit.

2. Training stability and longer training. Another line of work improves training stability through algorithmic innovations (EntroPIC, Yang et al., 2025b; MiniMax-M1, Chen et al., 2025a) or more efficient rollout strategies (selective rollouts, Zheng et al., 2025; difficulty prediction, Qu et al., 2025). These methods enable longer training without performance collapse, which is valuable, but again do not increase the effective dataset size. Longer training on a shrinking set of informative prompts eventually hits diminishing returns.

3. Data augmentation and synthesis. The most closely related work attempts to create new training prompts. Hint-based augmentation (Chen et al., 2025b; Li et al., 2025a) takes originally hard prompts and adds hints to make them easier, transforming solve_none prompts into informative ones. This is complementary to Composition-RL: it targets the hard end of the difficulty spectrum, while Composition-RL targets the easy end. Other work synthesizes entirely new prompts from scratch (Logic-RL, Xie et al., 2025; REASONING Gym, Stojanovski et al., 2025; InternBootcamp, Li et al., 2025b), but generating realistic, diverse math problems at scale is challenging—synthetic problems often lack the organic complexity and diversity of human-written datasets. Composition-RL sidesteps this by building new problems from existing ones, preserving their quality while increasing difficulty.

4. Exploration-focused methods. Knapsack RL (Li et al., 2025c) and Depth-Breadth Synergy (Yang et al., 2025c) allocate more rollouts to harder prompts, effectively increasing the exploration budget for problems the model struggles with. This helps convert some solve_none prompts into informative ones but still does not repurpose solve_all prompts.

The common thread is that prior work has overwhelmingly focused on making hard problems usable. The paper's key insight is that easy problems—which constitute an increasingly large fraction of the training set—are equally problematic and have been largely neglected.

How This Paper Positions Itself

Composition-RL is presented as a data-centric solution to the training data staleness problem, specifically targeting the solve_all bottleneck. Its intellectual lineage draws from two distinct areas:

Compositional generalization. The idea that models can learn to recombine known skills to solve novel problems has been studied extensively in NLP (Keysers et al., 2019; Hupkes et al., 2020; Lake and Baroni, 2018). More directly, Yuan et al. (2025) showed that in a synthetic string-transformation RL setting, training on composed examples (e.g., learning $f(g(x))$ from examples of $f(x)$ and $g(x)$ ) elicits new skills that individual training could not. Xiao and Zhao (2025) evaluated LLMs on composed math problems and found they struggle—performance drops substantially when two problems are chained. Composition-RL inverts this finding: rather than treating composition as an evaluation challenge, it uses composition as a training augmentation that forces the model to develop stronger reasoning.

Sequential Prompt Composition (SPC). The technical mechanism is a three-step procedure (Section 3.1) that chains two prompts: (1) modify $q_1$ to introduce a named variable $v_1$ representing a numeric value extracted from its answer, (2) modify $q_2$ by replacing one of its constants with a new variable $v_2$ , and (3) connect them with a constraint relating $v_1$ and $v_2$ . The composed prompt's answer is simply the answer to $q_2$ —but to solve it, the model must first solve $q_1$ to find $v_1$ , then use the constraint to compute $v_2$ , and finally solve the modified $q_2$ . This creates a structured dependency that is inherently verifiable (the final answer can still be checked against the ground truth) but substantially harder than either original problem.

The meta-experiment in Figure 2 (Right) validates this intuition: on a subset of MATH500, switching from original to composed (Depth-2) prompts reduces avg@8 accuracy by $15.4\%$ – $19.7\%$ and cuts the solve_all rate from $81.5\%$ – $88.5\%$ to $41.4\%$ – $60.0\%$ . Composed prompts are genuinely harder and re-activate previously stale training data.

Position relative to prior work. The paper explicitly frames Composition-RL not as a replacement for existing methods (better algorithms, exploration strategies, hint-based augmentation) but as a complementary approach that operates on an orthogonal axis—difficulty relocation (easy → hard) rather than difficulty mitigation (hard → easier, or zero-variance → non-zero advantage). The curriculum variant described in Section 4.3 makes this complementarity explicit: start with standard RL on original prompts (Depth 1), transition to compositional prompts when the solve_all rate plateaus (Depth 2), and continue to deeper compositions (Depth 3+). This integrates naturally with any existing RL algorithm and any existing prompt set.

The cross-domain extension. The paper also positions Composition-RL as a tool for cross-domain transfer. By composing a physics problem ( $q_1$ ) with a math problem ( $q_2$ ), the model must apply physics knowledge to find $v_1$ and then mathematical reasoning to solve $q_2$ —implicitly learning to bridge domains. The results in Section 4.4 and Table 2 show this is more effective than simply mixing physics and math prompts in the same batch or training sequentially on one domain then the other, suggesting that composition creates transfer benefits that naive data combination cannot.

Summary of the gap and the response. The paper identifies a neglected bottleneck (solve_all prompts wasting 75% of training data), shows that prompt composition effectively re-activates these prompts by making them harder, and demonstrates that this simple data transformation yields consistent, scaling-friendly gains across model sizes and domains—all without requiring new data collection, new RL algorithms, or new verifiers. The innovation is not in the RL method but in the recognition that easy prompts are a solvable problem if you can systematically make them harder while preserving verifiability.

3. Technical Approach

3.1 Reader Orientation

Composition-RL is a data augmentation pipeline that takes a set of existing verifiable training prompts—most of which have become trivially easy for the model and thus uninformative for RL—and automatically chains them together into harder compositional prompts that re-engage the model's learning process, without requiring any new data collection, changes to the RL algorithm, or modifications to the verifier.

The system solves the problem of training data staleness during RLVR: as the model improves, an increasing fraction of prompts become solve_all (all sampled responses correct), yielding zero advantage signals and contributing nothing to learning. The solution's shape is difficulty relocation via sequential prompt composition—take prompts the model has already mastered, modify them to depend on each other's answers, and thereby create a structured, harder prompt whose solution requires correctly executing a chain of reasoning steps, all while preserving the ability to verify the final answer against the original ground truth.

3.2 Big-Picture Architecture (Diagram in Words)

The Composition-RL system has three major stages, organized as a data construction pipeline followed by standard RL training:

Sequential Prompt Composition (SPC) Engine — an LLM-driven procedure that takes a pair of prompts $(q_1, q_2)$ with known ground-truth answers $(gt_1, gt_2)$ and produces a single composed prompt $q_{1:2}$ whose answer is $gt_2$ but whose solution requires first solving $q_1$ to recover an intermediate variable and then using that variable to solve a modified version of $q_2$ . This engine operates at scale to produce a compositional dataset $\mathcal{D}_{C_K}$ of arbitrary depth $K$ .
Verification and Filtering Pipeline — a three-step self-verification process that checks each composed prompt for internal consistency (whether the extracted variables match their sources, whether the connecting constraint is correct) and discards erroneous compositions, reducing the error rate below $2\%$ .
Standard RLVR Training Loop — the same GRPO objective, advantage estimator, dynamic sampling, and verifier (Math-Verify) used for training on original prompts, but now consuming compositional prompts from $\mathcal{D}_{C_K}$ instead of the original dataset $\mathcal{D}$ . The RL training itself is completely unchanged; only the data distribution differs.

Information flows as follows: the original prompt set $\mathcal{D}$ (e.g., MATH12K) enters the SPC engine → the engine composes pairs of prompts using Qwen2.5-32B-Instruct as the composition model → the verification pipeline filters out unreliable compositions → the surviving compositional prompts form $\hat{\mathcal{D}}_{C_K}$ → GRPO training proceeds identically to baseline RLVR, sampling groups of 8 responses per compositional prompt, scoring with Math-Verify, computing token-normalized advantages, and updating the policy.

3.3 Roadmap for the Deep Dive

First, the core composition operator Compose — how two prompts are mechanically combined into one, with precise definitions of the three sub-steps (modify $q_1$ , modify $q_2$ , connect them), since this is the atomic operation that makes everything else possible.
Second, the generalization to Sequential Prompt Composition (SPC) for arbitrary depth $K$ , including the recursive definition and the practical surrogate construction $\hat{\mathcal{D}}_{C_K}$ that makes scaling computationally feasible.
Third, the RLVR training objective and advantage computation — both because this is the consumer of compositional data and because understanding why solve_all prompts vanish (zero advantage) is essential to understanding why compositional data helps (it reduces the solve_all rate).
Fourth, the dynamic sampling filter — the mechanism that discards zero-advantage prompts and creates the data staleness problem that Composition-RL solves, along with the meta-experimental evidence quantifying the severity of the problem.
Fifth, the verification and filtering pipeline — how compositional data quality is maintained at scale, since feeding erroneous composed prompts into RL training would teach the model to rely on incorrect intermediate constraints.
Sixth, the curriculum extension — how Composition-RL is applied progressively across increasing compositional depths to sustain learning as the model saturates at each level.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data-centric methods paper whose core idea is that composing existing verifiable prompts into harder ones via structured chaining re-activates stale training data (specifically, solve_all prompts) and yields consistent RL performance gains without modifying the RL algorithm, verifier, or training data sources.

The Core Composition Operator: `Compose`

The atomic operation in Composition-RL is a function Compose that maps two prompts with known ground-truth answers to a single composed prompt and its answer:

$q_{1:2}, \, gt_{1:2} = \text{Compose}(q_1, q_2; \, gt_1, gt_2)$

where $q_1$ and $q_2$ are the original prompts, $gt_1$ and $gt_2$ are their ground-truth answers, $q_{1:2}$ is the composed prompt, and $gt_{1:2} = gt_2$ is the ground-truth answer of the composed prompt.

What it computes: the function takes two independently solvable math problems and produces a single new problem whose answer is identical to the second problem's answer, but whose solution requires first solving the first problem to obtain an intermediate value, then using that value (via a connecting constraint) to solve a modified version of the second problem.

Why this form: the asymmetric design—where the answer comes from $q_2$ but the solution depends on $q_1$ —creates a structural dependency that the verifier does not need to know about. The verifier only checks whether the final answer matches $gt_2$ , but the model must correctly execute the intermediate reasoning step (solving $q_1$ ) to determine the input to $q_2$ . This provides implicit process supervision: even though the reward signal only comes from the final answer, the composed structure naturally penalizes incorrect intermediate reasoning because an error in the first step propagates to a wrong final answer. This is fundamentally different from simply concatenating two problems and asking for both answers—there, the verifier would need to check two answers independently, and the model could get partial credit. Here, the chaining is lossy: if you fail $q_1$ , you fail everything.

The Compose operator is implemented in three sequential sub-steps, all automated using an LLM assistant (Qwen2.5-32B-Instruct with temperature $0.1$ , top_p $0.7$ , and maximum output length 4096 tokens, unless otherwise specified):

Sub-step ❶: Modify $q_1$ with $gt_1$

The first sub-step extracts a numeric value from $q_1$ 's answer and introduces it as a named variable. Specifically:

Extract a numeric value $v_1$ from $gt_1$ . The LLM identifies a meaningful numeric quantity in the answer to $q_1$ . For example, if $q_1$ is "What is the sum of the value(s) of $n$ for which $|2n - 7| = 3$ ?" and $gt_1 = 7$ , the extracted value is $v_1 = 7$ .
Introduce a natural-language definition $d_1$ . The LLM rewrites $q_1$ to include a definition that names the extracted value. Continuing the example, the definition might be: "Let $X$ be the sum of the value(s) of $n$ satisfying $|2n - 7| = 3$ ." The modified prompt becomes $\bar{q}_1 = q_1 \oplus d_1$ , where $\oplus$ denotes concatenation.

What this achieves: the original $q_1$ is transformed from a standalone question ("What is the sum?") into a context-setting preamble that defines a named variable ( $X$ ) whose value the model must compute. The model reading $\bar{q}_1$ knows that $X$ is a quantity it needs to determine, but the actual computation is identical to solving $q_1$ .

Design choice: the variable name ( $X$ , $Y$ , etc.) is chosen by the LLM to be semantically neutral but distinct. The authors do not use fixed naming conventions; the LLM has freedom to select variable names that fit naturally into the mathematical context. This introduces some natural variation in the compositional dataset, which may help prevent the model from overfitting to a specific composition template.

Sub-step ❷: Modify $q_2$

The second sub-step introduces a dependency on an unknown variable into $q_2$ . Specifically:

Extract a numeric value from $q_2$ and replace it with a new variable $v_2$ . The LLM identifies a constant in $q_2$ that can be abstracted into a variable. For example, if $q_2$ is "Simplify $2((5p + 1) - 2p \cdot 4) + (4 - 1 \div 3)(6p - 9)$ to the form $ap - b$ , where $a$ and $b$ are positive," the LLM might select the constant $1$ (appearing in $5p + 1$ ) and replace it with a variable name $Y$ , yielding $\bar{q}_2 = q_2(v_2)$ : "Simplify $2((5p + Y) - 2p \cdot 4) + (4 - 1 \div 3)(6p - 9)$ to the form $ap - b$ , where $a$ and $b$ are positive."

What this achieves: $q_2$ is no longer independently solvable—its answer depends on the unknown value $Y$ . The model cannot compute the final answer until it knows what $Y$ is. This creates the dependency that the third sub-step will resolve.

Design choice: the choice of which constant to replace matters. If the LLM replaces a constant that appears only once and has a simple linear effect on the answer, the resulting problem is straightforward once $Y$ is known. If it replaces a constant that appears in multiple places or has a non-linear interaction with other variables, the dependency is more complex. The paper does not control for this explicitly—the LLM selects the constant autonomously—but the verification step (see below) ensures the resulting composition is mathematically consistent.

Sub-step ❸: Connect $q_1$ and $q_2$

The third sub-step creates a numeric relationship between the two variables and concatenates everything:

Compute the relation between $v_1$ and $v_2$ . The LLM computes $v_1 - v_2$ (the paper uses subtraction as the default arithmetic operation) and expresses the result as a natural-language constraint $r$ . For example, with $v_1 = 7$ and $v_2 = 1$ , we have $v_1 - v_2 = 6$ , so the constraint is: " $Y$ is $6$ less than $X$ ."
Form the composed prompt. The final composed prompt is the concatenation: $q_{1:2} = \bar{q}_1 \oplus r \oplus \bar{q}_2$ . The ground-truth answer remains $gt_{1:2} = gt_2$ .

What this achieves: the composed prompt now tells a complete story: "Here is problem 1, which defines $X$ . $Y$ is related to $X$ by a known offset. Here is problem 2, which depends on $Y$ . Solve everything." To produce the final answer, the model must: (a) solve $\bar{q}_1$ to find $X$ , (b) apply the constraint $r$ to compute $Y = X - 6$ , and (c) plug $Y$ into $\bar{q}_2$ and solve the modified problem to get $gt_2$ .

Design choice: why subtraction? The paper uses subtraction ( $v_1 - v_2$ ) as the default way to relate the two variables, expressed in natural language (e.g., " $Y$ is $6$ less than $X$ "). Subtraction is sufficient because it covers all integer offsets: any difference can be expressed. However, this is not a hard constraint of the method—the LLM could in principle generate any arithmetic relationship, and the verification step would catch inconsistencies. Using subtraction makes the composition procedure deterministic given the extracted variables, which simplifies quality control at scale.

Why asymmetric composition? The composed prompt is fundamentally asymmetric: $q_1$ must be solved first, $q_2$ second. This is not an arbitrary choice—it mirrors how compositional generalization is studied in cognitive science and NLP (solving $f(g(x))$ requires solving $g$ first). The paper explicitly notes that the composition order matters and that swapping $q_1$ and $q_2$ would produce a different composed prompt (with answer $gt_1$ instead of $gt_2$ ). This asymmetry is a feature, not a bug: it means that from $N$ original prompts, the number of possible composed prompts is $N \times (N-1)$ (ordered pairs), not $N(N-1)/2$ (unordered pairs), providing more compositional variety per original prompt.

Generalization to Arbitrary Depth: Sequential Prompt Composition (SPC)

The Compose operator composes two prompts. SPC generalizes this to compose $K$ prompts by recursive application:

$\text{SPC}(q_1, \dots, q_K; \, gt_1, \dots, gt_K) = \text{Compose}(q_1, q_{2:K}; \, gt_1, gt_{2:K})$

where $(q_{2:K}, gt_{2:K})$ is itself the output of $\text{SPC}(q_2, \dots, q_K; \, gt_2, \dots, gt_K)$ , with the base case being $\text{SPC}(q_K; gt_K) = (q_K, gt_K)$ .

What it computes: SPC takes $K$ prompts with known answers and builds a single composed prompt by recursively chaining: the first prompt defines a variable that feeds into the composition of the remaining $K-1$ prompts. Depth $K = 1$ is the identity (original prompt). Depth $K = 2$ is the base Compose operation. Depth $K = 3$ composes the first prompt with a Depth-2 composition of the remaining two, and so on. The answer of the final composed prompt is always $gt_K$ .

What this enables: increasing $K$ creates progressively longer reasoning chains. A Depth-2 prompt requires two sequential sub-problem solutions. A Depth-3 prompt requires three. The curriculum variant (Section 4.3) exploits this: when the model saturates on Depth- $K$ compositions, the training data transitions to Depth- $(K+1)$ compositions, which are harder and re-activate the learning signal.

Practical limit: the paper experiments up to $K = 3$ . Beyond this, the composed prompts become extremely long (each original problem has its own context, and the composed prompt concatenates all of them plus connecting constraints), potentially exceeding the maximum output length of $16K$ tokens used during training. The paper does not explore $K \geq 4$ .

Surrogate Dataset Construction: $\hat{\mathcal{D}}_{C_K}$

In principle, composing every ordered $K$ -tuple from the original dataset $\mathcal{D}$ (with no repetition) yields a compositional dataset $\mathcal{D}_{C_K}$ of size $|\mathcal{D}| \times (|\mathcal{D}| - 1) \times \dots \times (|\mathcal{D}| - K + 1)$ . For MATH12K ( $|\mathcal{D}| \approx 12{,}000$ ) and $K = 2$ , this is approximately $144$ million prompts—far too many to generate, store, or train on.

The paper instead constructs a surrogate compositional dataset $\hat{\mathcal{D}}_{C_K}$ by restricting the candidate pools for each position:

$\hat{\mathcal{D}}_{C_K} = \{(q, gt) : q, gt = \text{SPC}(q_1, \dots, q_K; gt_1, \dots, gt_K), \, (q_k, gt_k) \in \mathcal{D}_k, \, k = 1, \dots, K, \, q_i \neq q_j \; \forall i \neq j\}$

where each $\mathcal{D}_k$ is a small random subset of $\mathcal{D}$ serving as the candidate pool for position $k$ . In practice:

$|\mathcal{D}_k| = 20$ for $k = 1, \dots, K-1$ (the first $K-1$ positions draw from small pools).
$\mathcal{D}_K = \mathcal{D}$ (the last position draws from the full dataset).

What this means concretely for $K = 2$ : $q_1$ is randomly sampled from a pool of $20$ prompts, and $q_2$ is randomly sampled from the full $12{,}000$ prompts. This yields at most $20 \times 12{,}000 = 240{,}000$ compositional prompts (fewer after filtering; see below). The compositional dataset used in the main experiments, MATH-Composition-199K, contains approximately $199{,}000$ prompts after the filtering pipeline.

Why this specific configuration? The ablation study in Section 5.1 (Table 3) answers this directly. The authors test three variants:

Variant A: Both $\mathcal{D}_1$ and $\mathcal{D}_2$ are small random subsets ( $|\mathcal{D}_1| = |\mathcal{D}_2| = 500$ ). This yields only $500 \times 500 = 250{,}000$ candidate compositions from just $1{,}000$ total seed prompts, substantially reducing diversity.
Variant B: $\mathcal{D}_1$ is the full set ( $|\mathcal{D}_1| = 12{,}000$ ) and $\mathcal{D}_2$ is a small subset ( $|\mathcal{D}_2| = 20$ ). This swaps which position gets the full pool.
Composition-RL (default): $\mathcal{D}_1$ is a small subset ( $|\mathcal{D}_1| = 20$ ) and $\mathcal{D}_2$ is the full set ( $|\mathcal{D}_2| = 12{,}000$ ).

The default configuration outperforms both alternatives. The authors hypothesize that the advantage comes from $\mathcal{D}_2$ being the full set: since the composed prompt's answer is $gt_2$ (the answer to the second prompt), using the full dataset for $\mathcal{D}_2$ maximizes the diversity of final answers the model must learn to produce. Under Variant B, the model is repeatedly trained on only $20$ distinct answers (those from the small $\mathcal{D}_2$ ), which limits the coverage of training signals. Under Variant A, both prompt diversity and answer diversity are reduced.

The critical insight is that the last position in the composition determines the answer, so it should draw from the full dataset to maximize answer diversity. The earlier positions determine the intermediate variables but not the final answer, so they can use smaller pools without severely limiting the training signal.

RLVR Training Objective: Why Compositional Data Helps

To understand why compositional data improves RL training, we must first understand why the original data stops providing useful gradients. The RLVR objective maximizes expected verifiable reward:

$\max_\theta \; \mathbb{E}_{q \sim \mathcal{D}} \left[ \mathcal{J}_{\text{RLVR}}(\theta, q) \right] = \mathbb{E}_{q \sim \mathcal{D}, \, r \sim \pi_\theta(\cdot \mid q)} \left[ v(q, r) \right]$

where $\theta$ are the policy parameters, $q$ is a prompt from the dataset $\mathcal{D}$ , $r$ is a response sampled from the policy $\pi_\theta$ , and $v(q, r) \in \{0, 1\}$ is the verifier's binary judgment of whether $r$ 's final answer matches the ground truth.

The policy gradient for a single response is:

$g_\theta(q, r) = A(q, r) \cdot \nabla_\theta \log \pi_\theta(r \mid q)$

where $A(q, r) = v(q, r) - b(q)$ is the advantage—the verifier score minus a baseline $b(q)$ that depends only on the query. The advantage determines both the magnitude and the direction of the gradient update.

What these equations compute: the RLVR objective says "sample prompts, sample responses, score them, and move the policy parameters in the direction that increases the log-probability of high-scoring responses and decreases the log-probability of low-scoring ones." The advantage $A(q, r)$ acts as a weight: if $A$ is positive, the gradient pushes the policy to make $r$ more likely; if negative, less likely; if zero, the gradient vanishes entirely.

GRPO advantage estimation (Equation 2):

$\hat{A}_i = \frac{v(q, r_i) - \text{mean}(\{v(q, r_j)\}_{j=1}^G)}{\text{std}(\{v(q, r_j)\}_{j=1}^G)}$

where $G$ is the group size ( $8$ in all experiments), ${r_1, \dots, r_G}$ are $G$ responses sampled from the old policy $\pi_{\theta_{\text{old}}}$ for the same prompt $q$ , and $\hat{A}_i$ is the estimated advantage for response $i$ .

What this computes: the GRPO advantage is a z-score normalization within the group of $G$ responses. It subtracts the mean verifier score across the group and divides by the standard deviation. If all responses are correct ( $v(q, r_j) = 1$ for all $j$ ), then $\text{mean} = 1$ , $\text{std} = 0$ , and $\hat{A}_i = 0$ for all $i$ (division by zero is handled by setting advantage to zero). Similarly, if all responses are incorrect ( $v(q, r_j) = 0$ for all $j$ ), $\text{mean} = 0$ , $\text{std} = 0$ , and again $\hat{A}_i = 0$ for all $i$ .

Why this form matters for understanding Composition-RL: the GRPO advantage fundamentally requires within-group variance to produce non-zero signals. A solve_all prompt has zero variance—every response is correct, so there is nothing to learn from. A solve_none prompt also has zero variance. Only prompts where some responses are correct and some are incorrect produce non-zero advantages. This is the mechanism by which easy prompts become useless: as the model improves, more prompts become solve_all, and the effective training set shrinks.

The token-normalized GRPO objective (Equation 3):

$\frac{1}{\sum_{i=1}^G |r_i|} \sum_{i=1}^G \sum_{t=1}^{|r_i|} \min\left( i_{i,t}(\theta) \hat{A}_i, \; \text{clip}(i_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_i \right)$

where $i_{i,t}(\theta)$ is the token-level importance ratio (Equation 4):

$i_{i,t}(\theta) = \frac{\pi_\theta(r_{i,t} \mid q, r_{i,<t})}{\pi_{\theta_{\text{old}}}(r_{i,t} \mid q, r_{i,<t})}$

What this computes: this is the standard PPO-style clipped surrogate objective applied at the token level, with the sum normalized by the total number of tokens across all responses in the group. The importance ratio $i_{i,t}(\theta)$ measures how much the new policy's probability for token $r_{i,t}$ has changed relative to the old policy's probability. The clip operation prevents the update from being too large when the ratio deviates far from 1. The $1 / \sum |r_i|$ normalization ensures that longer responses do not dominate the objective simply because they have more tokens.

Why token-level normalization matters for compositional data: compositional prompts typically require longer responses (the model must solve multiple sub-problems sequentially), so the token-level normalization ensures that training on compositional prompts is fairly weighted relative to training on original prompts. Without this normalization, compositional prompts would contribute disproportionately to the gradient simply because they produce longer responses, not because they are more informative.

Dynamic Sampling: Where the Data Staleness Problem Becomes Visible

The paper uses dynamic sampling (Equation 5) to filter uninformative prompts from each training batch:

$\mathcal{B} = \left\{ q \in \hat{\mathcal{B}} : 0 < \text{mean}(\{v(q, r_j)\}_{j=1}^G) < 1 \right\}$

where $\hat{\mathcal{B}}$ is an over-sampled candidate batch (more prompts than needed for the actual training batch) and $\mathcal{B}$ is the final training batch after filtering out prompts where all $G$ responses are either all correct or all incorrect.

What this computes: for each prompt in the over-sampled set, the system samples $G = 8$ responses, scores them, computes the mean score, and keeps only prompts where the mean is strictly between $0$ and $1$ . Prompts with mean $0$ (all incorrect, solve_none) or mean $1$ (all correct, solve_all) are discarded. The training batch $\mathcal{B}$ is then formed from the surviving prompts, with the batch size kept constant at $256$ by over-sampling enough prompts to compensate for the discarded fraction.

Why this filter reveals the solve_all problem: the filter's purpose is to save computation—there is no point computing gradients for prompts that would yield zero advantage. But the side effect is that the effective training set size is the number of prompts that pass the filter. The meta-experiment in Figure 2 (Left) measures this directly: during standard RL on MATH12K, the solve_all rate rises to approximately $75\%$ by step $250$ , meaning only about $12{,}000 \times 0.25 = 3{,}000$ prompts pass the filter. The solve_none rate remains low (about $5\%$ ), so the vast majority of discarded prompts are discarded because they are too easy, not too hard.

How compositional data changes this: the meta-experiment in Figure 2 (Right) evaluates pre-trained reasoning models (OpenMath-Reasoning-1.5B and JustRL-1.5B) on original vs. compositional prompts. On the original MATH500 subset, the solve_all rate is $81.5\%$ – $88.5\%$ —nearly $9$ out of $10$ prompts are trivially easy. On the compositional test set, the solve_all rate drops to $41.4\%$ – $60.0\%$ . The compositional prompts are substantially harder, meaning more of them will fall into the informative middle range where the filter keeps them. This is the core mechanism by which Composition-RL re-activates stale data: prompts that were solve_all in their original form become mixed-difficulty when composed, producing non-zero advantages and contributing gradient signals again.

The SPC Implementation and Verification Pipeline

The entire SPC process is automated using Qwen2.5-32B-Instruct as the composition model. The prompts used for each sub-step and for verification are provided in Appendix D.3 (Figures 5 and 6). To ensure reliability at scale—since LLM-generated compositions can contain errors—the pipeline includes three verification steps, each using the same LLM for self-consistency checking:

Verification ❶: Check the modification of $q_1$ . After Step ❶ generates $\bar{q}_1$ with variable $v_1$ and definition $d_1$ , the LLM is prompted to re-solve the problem described in $\bar{q}_1$ : given the original $q_1$ and the definition $d_1$ , what is the value of $v_1$ ? If the LLM's computed value does not match the extracted $v_1$ , the composition is discarded. This catches cases where the LLM extracted the wrong numeric value from $gt_1$ or introduced a definition that is inconsistent with $q_1$ .

Verification ❷: Check the modification of $q_2$ . Similarly, the LLM verifies that the extracted variable $v_2$ and its replacement are consistent with $q_2$ : does the modified problem $\bar{q}_2$ correctly abstract the constant that was replaced? If the LLM replaced the wrong constant (e.g., one that doesn't actually appear in $q_2$ ) or introduced a variable that makes the problem ill-defined, this check catches it.

Verification ❸: Check the overall composed prompt. After Step ❸ concatenates everything, the LLM checks for inconsistencies: conflicting variable names, arithmetic errors in the constraint $r$ , or logical gaps that would make the composed prompt unsolvable.

Filtering yield. The paper reports the yield at each stage for MATH12K with the default configuration ( $|\mathcal{D}_1| = 20$ , $\mathcal{D}_2 = \mathcal{D}$ ):

After Verification ❶: approximately 231K prompts survive (from the theoretical maximum of 240K).
After Verification ❷: approximately 200K survive.
After Verification ❸: approximately 199K survive (MATH-Composition-199K).

The total filtering removes about $17\%$ of candidate compositions, primarily at the first verification step. The paper cites Xiao and Zhao (2025) for the claim that the final error rate in the surviving prompts is below $2\%$ , which the authors consider acceptable for training.

Why this verification is necessary: the composition procedure relies on an LLM to extract variables, rewrite problems, and generate constraints—all operations that the LLM can perform imperfectly. If erroneous compositions entered the training set, the model would be trained on prompts with incorrect ground-truth relationships (e.g., a constraint that says " $Y$ is $6$ less than $X$ " when the true difference is $5$ ). This would teach the model to produce wrong intermediate computations, undermining the implicit process supervision benefit. The verification pipeline trades off a small amount of data loss for substantially higher data quality.

Curriculum Composition-RL: Progressive Depth Increase

The curriculum variant described in Section 4.3 is not a separate algorithm but a training schedule that transitions between compositional depths as performance plateaus. The procedure is:

Depth 1 (standard RL): Train on the original MATH12K prompts until the solve_all rate stabilizes at a high level and validation performance (e.g., on AIME24) plateaus. This is the "Depth 1 → Depth 2" checkpoint.
Depth 2 (first composition): Switch the training data to MATH-Composition-199K (Depth-2 compositions) and continue training. The solve_all rate drops sharply because the compositional prompts are harder, and validation performance resumes improving.
Depth 3 (second composition): Once Depth 2 saturates, switch to Depth-3 compositions (constructed analogously to Depth 2, by composing an additional prompt onto Depth-2 compositions). Training continues, and performance improves further.

Why a curriculum rather than direct Depth-3 training? The paper does not ablate this directly, but the logic follows from the nature of compositional difficulty: a model initialized from a base checkpoint (e.g., Qwen3-4B-Base) may not have the capability to solve Depth-3 compositions at all—producing only solve_none responses, which are also uninformative. By first training on Depth 1 (original prompts) to build basic math reasoning skills, then Depth 2 (two-problem chains) to build composition skills, and finally Depth 3, the model always faces prompts at the frontier of its capability. This is an instance of curriculum learning applied to compositional depth, where the curriculum is defined by the number of chained sub-problems.

The observed benefit: the curriculum variant (Depth 1 → Depth 2) outperforms direct training on MATH-Composition-199K (Depth 2 only) by an additional $+3.0\%$ overall average. This suggests that the skills learned from original prompts transfer to compositional prompts (consistent with the observation in Figure 2, Right, that RL on normal prompts improves compositional performance), and that building on this transfer yields better results than starting from compositional data directly.

Cross-Domain Composition: Physics-MATH-Composition-141K

The cross-domain extension in Section 4.4 applies the same SPC procedure but with $q_1$ drawn from the physics subset of MegaScience (approximately $8.2K$ prompts after filtering for rule-based verifiability) and $q_2$ drawn from MATH12K. The resulting dataset, Physics-MATH-Composition-141K, contains composed prompts where solving requires: (a) applying physics knowledge to compute $v_1$ from a physics problem, (b) using the connecting constraint to determine $v_2$ , and (c) using mathematical reasoning to solve the modified math problem with $v_2$ as input.

Why the asymmetric composition order (physics → math)? The paper does not explicitly justify this choice, but it follows from the structure of the Compose operator: $q_1$ defines a variable that feeds into $q_2$ , and $q_2$ determines the final answer. By putting physics first and math second, the composition tests whether physics knowledge can be integrated into mathematical problem-solving—a natural cross-domain transfer scenario. The reverse order (math → physics) would have the answer be a physics result, which may be less useful for the evaluation benchmarks (which heavily weight math reasoning).

The filtering for physics data: because Math-Verify (a rule-based verifier designed for math problems) does not reliably evaluate physics problem solutions, the authors pre-filter the MegaScience physics subset by removing any prompt where all 8 responses from Qwen3-4B-Thinking-2507 are judged incorrect by Math-Verify. This yields approximately $8.2K$ prompts where the verifier's judgments are trustworthy. This is a pragmatic constraint: Composition-RL relies on the same verifier used for standard RLVR, so all composed prompts must be verifiable by Math-Verify. If the physics problem's answer cannot be reliably checked, any composed prompt that includes it cannot be verified either.

Summary of Design Choices and Their Justifications

Asymmetric composition ( $q_1$ feeds into $q_2$ , answer is $gt_2$ ): creates a structured dependency that provides implicit process supervision—incorrect intermediate reasoning causes the final answer to be wrong, even though the verifier only checks the final answer. This is the core mechanism that makes compositional data more informative than random concatenation.
Small candidate pool for $q_1$ ( $|\mathcal{D}_1| = 20$ ), full dataset for $q_2$ ( $\mathcal{D}_2 = \mathcal{D}$ ): the last position determines the answer, so using the full dataset for it maximizes answer diversity and prevents the model from overfitting to a small set of final answers. The first position only determines an intermediate variable, so a small pool provides sufficient variety.
LLM-based composition with three-step verification: fully automated pipeline that scales to hundreds of thousands of prompts without human curation, while maintaining error rates below $2\%$ through self-consistency checks.
Subtraction ( $v_1 - v_2$ ) as the default connecting operation: simple and deterministic given the extracted variables, making quality control straightforward. More complex relationships could be generated but would require more sophisticated verification.
Curriculum over compositional depth rather than direct deep composition: ensures the model always trains at the frontier of its capability, avoiding the dual failure modes of solve_all (too easy) and solve_none (too hard).
No modifications to the RL algorithm or verifier: Composition-RL is purely a data transformation; it plugs into any existing RLVR pipeline without changing the training code, advantage computation, or verification logic. This makes it immediately applicable to existing RLVR setups with any algorithm (GRPO, PPO, REINFORCE) and any verifier.

4. Key Insights and Innovations

Innovation 1: Easy Prompts Are the Neglected Bottleneck in RLVR Training Data, Not Just Hard Ones

The paper's most important conceptual move is a diagnostic reframing of the RLVR data bottleneck. Prior work on extending RL training with finite prompts has focused almost entirely on the hard-prompt problem—how to extract useful gradient signals from solve_none prompts where all sampled responses are incorrect. This line of work has produced sophisticated solutions: entropy-guided advantage shaping (RL-ZVP, Le et al., 2025), negative reinforcement (NGRPO, Nan et al., 2025), high-entropy token weighting (Beyond-80/20, Wang et al., 2025), and hint-based augmentation that makes hard prompts easier (Chen et al., 2025b; Li et al., 2025a). Each of these targets a different mechanism for salvaging gradients from prompts the model cannot yet solve.

Composition-RL identifies a different and arguably larger bottleneck that prior work essentially ignored: solve_all prompts, where all sampled responses are correct. The meta-experiment in Figure 2 (Left) is the diagnostic punchline—the solve_all rate on MATH12K rises to over 50% within 50 training steps and stabilizes around 75%, meaning only ~3K of 12K prompts remain informative at later training stages. The solve_none rate, by contrast, stays around 5%. This means the dominant mechanism of data loss during RLVR is not that problems are too hard, but that they become trivially easy—and since a binary verifier produces zero advantage variance when all responses score identically, these prompts contribute nothing regardless of the RL algorithm.

This is a genuine reframing, not an incremental finding, because it inverts the default assumption in the RLVR literature. The field has implicitly treated solve_all prompts as a sign of success (the model has mastered the problem) and solve_none prompts as the training bottleneck to engineer around. Composition-RL argues that solve_all prompts are equally wasteful and constitute the majority of wasted data in the later stages of training—a claim substantiated by the solve_all curves but not previously articulated as a first-class problem requiring its own solution. The distinction between "the model knows this" (solve_all) and "the model has nothing to learn from this" (zero advantage) is subtle but practically devastating: a prompt the model consistently solves still carries zero information for policy improvement under standard GRPO advantage estimation.

The significance extends beyond this paper: if solve_all prompts are the dominant bottleneck, then the entire RLVR data scaling agenda (collecting bigger datasets, synthesizing new prompts, improving training stability for longer runs) is attacking a problem that is only partially understood. The inference-time compute scaling paper (the reference example in this prompt) showed that test-time strategies are difficulty-dependent; Composition-RL shows that training-time data utility is also difficulty-dependent, and that the failure mode at the easy end has been systematically overlooked.

The paper ties this reframing to concrete evidence: Figure 2 (Right) shows that composed prompts reduce the solve_all rate from 81.5–88.5% to 41.4–60.0% on pre-trained reasoning models, directly demonstrating that composition re-activates stale data. The curriculum variant (Figure 1, bottom-left) shows that transitioning from Depth 1 to Depth 2 prompts causes a sharp solve_all drop and renewed performance improvement—a direct operationalization of the diagnostic insight.

Innovation 2: Prompt Composition as Implicit Process Supervision Without a Learned Verifier

The second conceptual contribution is revealing that structured prompt composition can provide process-level supervision using only outcome-level verification. This is an insight about the relationship between prompt structure and training signal, not about a new RL algorithm or verifier architecture.

The mechanism (described in Section 3) is straightforward: compose two prompts such that the answer depends on a chain of intermediate computations (solve $q_1$ , extract $v_1$ , apply constraint, compute $v_2$ , solve $q_2$ ). The verifier only checks the final answer against $gt_2$ , but the model cannot produce the correct final answer without executing the intermediate steps correctly. An error in the first sub-problem propagates to a wrong final answer, so the outcome reward carries implicit information about the correctness of the intermediate reasoning—even though the verifier never explicitly scores those steps.

This is fundamentally different from how process supervision is typically achieved in RLVR. The standard approach—epitomized by process reward models (PRMs) trained with Monte Carlo rollouts (Lightman et al., 2023; Wang et al., 2023; the reference example paper's Section 5.1)—requires training a separate model to score intermediate steps, which is expensive, requires on-policy data, and introduces verifier over-optimization risks. Composition-RL achieves a qualitatively similar effect (the model receives gradient signals that incentivize correct intermediate reasoning) without any verifier modification whatsoever—the same binary rule-based verifier that checks final answers is sufficient.

The evidence for this claim rests on Figure 3 (Right), which shows that during Composition-RL training, the model's accuracy at recovering the intermediate variable $v_1$ (the value extracted from $q_1$ 's answer) steadily improves over training steps. This cannot be attributed to the verifier directly rewarding $v_1$ recovery (it never sees $v_1$ ), so it must arise from the structured dependency in the composed prompt: the model learns that getting $v_1$ wrong leads to the wrong final answer, and gradient signals from incorrect final answers push the policy toward correct intermediate computation.

This is not a theoretical advance in the sense of proving new properties of gradient estimators, but it is a conceptual contribution about data design: it demonstrates that there exists a class of prompt transformations that convert outcome-only verification into effective process-level training signals, without requiring the expensive machinery of separate verifier training. The significance is practical (it removes a major barrier to process-supervision-like benefits) but also conceptual: it suggests that the distinction between "outcome supervision" and "process supervision" is not purely a property of the verifier—it can be induced through the structure of the prompt itself when that structure forces intermediate reasoning to be causally necessary for the final answer.

The contrast with the reference example paper is instructive. That paper showed that PRMs are useful for test-time search but over-optimize at high budgets; Composition-RL shows that you can get some PRM-like benefits during training without training a PRM at all, simply by designing prompts that make intermediate steps causally load-bearing. This is a different point on the same design space: instead of building better verifiers, build better prompts.

Innovation 3: Compositional Difficulty as a Controllable, Reusable Curriculum Axis Independent of Prompt Collection

The third innovation is establishing compositional depth as a learnable curriculum dimension that can be adjusted dynamically during training, entirely independently of the underlying prompt content, enabling sustained learning from a fixed pool of original prompts. This is a meta-learning insight about how to structure the relationship between data and iterative training, not about any single composition trick.

Prior work on curriculum learning for LLM reasoning has typically relied on difficulty labels (assigning problems to difficulty levels based on some metric) or dynamic sampling (filtering prompts based on current model performance). These approaches are inherently consumption-based: they select among existing prompts but do not transform them. When all prompts in a given difficulty range become solve_all, the curriculum has no more material to offer at that level.

Composition-RL introduces a fundamentally different curriculum mechanism: increasing the compositional depth of the same underlying prompts creates an automatic difficulty escalation that is orthogonal to the content of individual problems. A prompt that is trivially easy at Depth 1 (standalone) becomes challenging at Depth 2 (composed with another problem) and potentially insurmountable at Depth 3. The curriculum is not "start with easy problems, graduate to hard problems" but rather "start with standalone problems, graduate to chains of 2 problems, then chains of 3"—the underlying content pool remains fixed, but the structural complexity of prompts drawn from it increases.

This is significant for two reasons. First, it decouples difficulty scaling from data scaling. Standard approaches require either collecting harder prompts (expensive) or filtering existing prompts for difficulty (which caps out when the hardest prompts are mastered). Composition-RL can generate arbitrarily deep compositions (bounded only by context length constraints), theoretically enabling difficulty scaling well beyond the difficulty range of the original dataset. The paper demonstrates up to Depth 3, but the principle extends further.

Second, the curriculum is content-preserving in a specific sense: the model never stops practicing the underlying skills (solving the original problems) because each composed prompt still requires executing those skills as sub-routines. This is unlike a typical curriculum that would discard easy content in favor of hard content—here, the easy content is embedded within the hard content, providing continual reinforcement of basic capabilities even as compositional complexity increases.

The evidence structure supports this as a genuine innovation rather than mere hyperparameter tuning. Figure 1 (bottom-left) shows that the Depth 1 → Depth 2 curriculum produces a sharp solve_all drop and renewed performance improvement each time compositional depth increases, and outperforms direct Depth 2 training by +3.0% overall. Table 1 shows that Depth 1 → Depth 2 → Depth 3 continues to improve, with the 4B model ultimately reaching 37.9% on AIME24—competitive with 8B models trained on larger datasets. The fact that a 4B model trained only on MATH12K (with compositional depth escalation) can match or exceed 8B models trained on DAPO-MATH-17K using sophisticated RL algorithms is strong circumstantial evidence that the curriculum mechanism itself is doing meaningful work beyond what mere data augmentation would provide.

This is a fundamental shift in how to think about RLVR data: rather than a fixed pool that gets consumed, the training set becomes a generative system where prompts can be recombined at increasing complexity levels, with the curriculum defined by the depth of recombination rather than by external difficulty labels. It opens the door to RLVR training regimes where data complexity grows in lockstep with model capability, potentially breaking the scaling bottleneck identified in Innovation 1.

Innovation 4: Cross-Domain Composition as a Data-Driven Alternative to Multi-Task or Sequential Training

The fourth innovation is demonstrating that composing prompts across domains is more effective for transfer than standard multi-domain training strategies—both mixed training (concatenating datasets) and sequential training (training on one domain, then another). This is not just a performance result; it reveals something about the nature of cross-domain transfer in RLVR that standard training paradigms fail to exploit.

The standard approaches to multi-domain RLVR are well-established: either mix all domains in the same training batch (Mix Training) to learn a shared policy, or train sequentially (domain A, then domain B) to avoid catastrophic interference while still achieving transfer. Both are methods for exposing the model to multiple data distributions, but they treat domains as separate—the model either sees physics problems and math problems interleaved in the same batch, or it sees them in sequence, but never sees a single problem that requires both simultaneously.

Composition-RL creates a third category: integrated prompts where solving requires applying knowledge from domain A (physics, to compute $v_1$ ) and domain B (math, to compute the final answer from $v_2$ ) within the same reasoning chain. This is qualitatively different from seeing both domains separately because it forces the model to bridge them: the physics computation must produce a value that the math computation consumes, creating a causal link between the two knowledge systems.

The results in Table 2 and Figure 1 (bottom-right) substantiate the claim. Physics-MATH-Composition-141K outperforms Mix Training (+5.8% on AIME24, +2.1% on MMLU-Pro) and Math-then-Physics (+7.1% on AIME24, +1.3% on MMLU-Pro). Notably, Mix Training actually degrades math performance relative to math-only training (16.6% → 14.1% overall math average), while compositional training improves it (16.6% → 21.6%). This means composition doesn't just avoid the interference that mixture training causes—it actively creates positive transfer that neither mixture nor sequential training achieves.

This is an insight about the structure of transferable knowledge: the model doesn't just need to know physics and math separately; it needs to know how to use physics results as inputs to math reasoning. Standard multi-domain training leaves this integration skill to emerge implicitly (if at all) from seeing both domains independently. Compositional training teaches it explicitly by constructing prompts where integration is required to succeed. This connects to the compositional generalization literature (Yuan et al., 2025; Zhao et al., 2024) but extends it from within-domain skill composition to cross-domain knowledge composition—a substantially harder problem that the paper shows is tractable via the same SPC mechanism.

The significance extends beyond math and physics. Any pair of domains where one domain's outputs can serve as another domain's inputs (code generation + test case execution, data analysis + visualization, fact retrieval + reasoning) could potentially benefit from cross-domain composition. The paper provides a template and a proof of concept, establishing that composition is not just a difficulty-escalation tool (Innovation 3) but also a knowledge-integration tool—and that the two uses can be pursued simultaneously with the same SPC pipeline.

This is an incremental contribution relative to the core SPC mechanism (which was already defined for within-domain composition), but it is a significant expansion of scope that transforms Composition-RL from a math-specific data augmentation trick into a general framework for cross-domain RLVR training. The fact that it works without any modification to the composition procedure—the same three-step SPC used for math-math compositions applies unchanged to physics-math compositions, with the LLM handling domain-specific variable extraction and constraint generation automatically—is noteworthy in its own right, suggesting the approach may generalize broadly with minimal adaptation.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary training dataset is MATH12K (the MATH training set from Hendrycks et al., 2021, with the MATH500 test set excluded, leaving approximately 12,000 prompts spanning five difficulty levels). For cross-domain experiments (Section 4.4), the authors use the physics subset of MegaScience (Fan et al., 2025), filtered to approximately 8.2K prompts where Math-Verify produces reliable judgments (removing prompts where all 8 responses from Qwen3-4B-Thinking-2507 are scored incorrect). The compositional datasets are MATH-Composition-199K (~199K Depth-2 prompts, constructed from MATH12K using the SPC procedure with $|\mathcal{D}_1| = 20$ , $\mathcal{D}_2 = \mathcal{D}$ , and the three-step verification pipeline) and Physics-MATH-Composition-141K (~141K prompts, with $q_1$ drawn from the physics subset and $q_2$ from MATH12K). Evaluation benchmarks include in-domain math tasks (AIME24, AIME25, BeyondAIME, IMOBench) and out-of-domain multi-task reasoning tasks (GPQA-Diamond, MMLU-Pro).
Base model(s). The experiments use the Qwen3 family of base models: Qwen3-4B-Base, Qwen3-8B-Base, Qwen3-14B-Base, and Qwen3-30B-A3B-Base (a Mixture-of-Experts model with 30B total parameters and 3B active parameters). These models are chosen to span a range of scales (4B to 30B) representative of contemporary open-weight LLMs, with a mix of dense and MoE architectures. For the composition pipeline (SPC and verification), the paper uses Qwen2.5-32B-Instruct as the assistant model. For the meta-experiment in Figure 2 (Right), the authors evaluate OpenMath-Reasoning-1.5B and JustRL-1.5B (a model obtained by RL-training the former) to provide initial evidence that compositional prompts are harder than original prompts.
Metrics. The primary metric throughout is pass@1 accuracy (%) — the fraction of problems for which a single sampled response (or the average over $k$ samples, denoted avg@k) produces the correct answer as judged by the verifier. For AIME24 and AIME25 (30 problems each), the paper reports avg@32 (accuracy averaged over 32 independent samples per problem). For BeyondAIME (100 problems), avg@8. For IMOBench (400 problems, AnswerBench subset), avg@4. For GPQA-Diamond (~200 problems), avg@8. For MMLU-Pro (>5K problems), pass@1 from a single run. The paper also reports overall mathematics (average across the four math benchmarks), overall multi-task (average across GPQA-Diamond and MMLU-Pro), and overall average (average across all six benchmarks). The meta-experiments track the solve_all ratio — the fraction of prompts for which all 8 sampled responses are correct — as a diagnostic for how much of the training data remains informative. All evaluation decoding uses temperature 0.6, top_p 0.95, top_k 20, and maximum output length 32K tokens, with vLLM for accelerated inference and Math-Verify for answer checking.
Baselines. For the main Composition-RL experiments (Section 4.2, Table 1), the key baseline is standard RLVR on MATH12K — GRPO training on the original 12K MATH prompts with dynamic sampling enabled, using the same hyperparameters, batch size (256), learning rate ( $1 \times 10^{-6}$ ), group size (8), and number of gradient updates as Composition-RL. For the curriculum experiments (Section 4.3), three additional RL-zero baselines from prior work are reported: Beyond-80/20 (Wang et al., 2025), AlphaRL (Cai et al., 2025), and RL-ZVP (Le et al., 2025). These baselines train from Qwen3-8B-Base on DAPO-MATH-17K and are quoted directly from the respective papers since the models were not publicly released; the paper acknowledges this comparison is unfavorable to Composition-RL (4B vs. 8B base, MATH12K vs. DAPO-MATH-17K). For the cross-domain experiments (Section 4.4, Table 2), two baselines are compared: Mix Training (RL on a dataset that concatenates MATH12K and the MegaScience physics subset) and Math-then-Physics (continued RL on physics data starting from a MATH12K-trained checkpoint), with both trained for approximately the same total number of gradient updates.
Generation budget / compute accounting. The paper does not use a separate generation budget metric; instead, training is matched on number of gradient updates — Composition-RL and the baseline RLVR training are run for the same number of steps, with the same batch size (256), same group size (8 rollouts per prompt), and same dynamic sampling filter. This ensures that any performance difference is attributable to the data distribution (compositional vs. original prompts), not to differences in total computation. For each compositional dataset, the total number of available prompts ( $\sim$ 199K for MATH-Composition-199K, $\sim$ 141K for Physics-MATH-Composition-141K) is large enough that the model never sees the same prompt repeatedly, matching the effective data diversity condition of the baseline.
Cross-validation / statistical protocol. The paper does not use cross-validation or statistical significance testing. Results are reported as point estimates from single training runs. The curriculum variant uses a heuristic saturation criterion — transitioning from Depth 1 to Depth 2 to Depth 3 when validation performance (tracked during training, as shown in Figure 1, bottom-left) plateaus — but no formal stopping rule or statistical test is specified. The ablation study in Section 5.1 reports results from training runs with matched dataset sizes and gradient updates. The meta-experiment in Figure 2 (Left) plots the solve_all ratio over training steps; the right panel uses avg@8 on a 200-question subset of MATH500 with approximately 400 compositional test prompts. All evaluation benchmarks use fixed random seeds for the sampling runs (32 samples for AIME, 8 for BeyondAIME, etc.), but the paper does not report confidence intervals or standard errors for any of the reported numbers.

Main Quantitative Results

Composition-RL vs. Standard RLVR on Original Prompts (Section 4.2, Table 1)

Headline result: Composition-RL consistently outperforms standard RLVR on MATH12K across all four model scales on both in-domain mathematics and out-of-domain multi-task benchmarks, with overall average gains of +3.3%, +3.7%, +4.3%, and +10.5% for the 4B, 8B, 14B, and 30B-A3B models respectively.

Detailed model-by-model breakdown (Table 1):

Qwen3-4B-Base: Composition-RL improves overall math average from 23.8% to 27.4% (+3.6%), with notable gains on AIME24 (19.9% → 24.4%, +4.5%), AIME25 (15.0% → 20.4%, +5.4%), and IMOBench (18.0% → 23.0%, +5.0%). Multi-task improves from 50.1% to 52.8% (+2.7%), driven by both GPQA (40.4% → 43.8%, +3.4%) and MMLU-Pro (59.8% → 61.8%, +2.0%). Overall average: 34.1% → 37.4% (+3.3%).
Qwen3-8B-Base: Math improves from 34.9% to 39.7% (+4.8%), with larger absolute gains on harder benchmarks: AIME24 (33.4% → 40.9%, +7.5%), BeyondAIME (13.2% → 23.1%, +9.9%), IMOBench (22.0% → 26.3%, +4.3%). Multi-task improves from 55.7% to 57.0% (+1.3%). Overall average: 44.0% → 47.7% (+3.7%).
Qwen3-14B-Base: Math improves from 41.6% to 47.7% (+6.1%), with AIME24 (40.1% → 50.7%, +10.6%) and AIME25 (32.0% → 39.8%, +7.8%) showing the largest gains. Multi-task improves from 58.3% to 59.0% (+0.7%). Overall average: 49.1% → 53.4% (+4.3%).
Qwen3-30B-A3B: Math improves from 28.5% to 42.8% (+14.3%), with massive gains on AIME24 (29.6% → 51.0%, +21.4%), AIME25 (20.7% → 34.8%, +14.1%), and BeyondAIME (10.5% → 22.5%, +12.0%). Multi-task improves from 47.2% to 50.1% (+2.9%). Overall average: 36.1% → 46.6% (+10.5%). Notably, the MoE 30B-A3B model underperforms the 14B dense model in absolute terms (baseline average 36.1% vs. 49.1%), consistent with the fact that MoE activates only a subset of experts per token and can be more sensitive to optimization under a fixed training budget. However, Composition-RL yields substantially larger relative gains on the MoE architecture, nearly closing the gap to the dense 14B model.

Key patterns across model sizes:

Gains increase with model scale: the overall average improvement grows from +3.3% (4B) to +10.5% (30B-A3B), with in-domain math improvements scaling even more sharply (+3.6% → +14.3%). This suggests that larger models have greater capacity to absorb the more complex reasoning patterns required by compositional prompts, and that the benefits of compositional data compound with parameter count.
OOD gains are smaller but consistently positive: multi-task improvements range from +0.7% (14B) to +2.9% (30B-A3B), suggesting that compositional math training transfers modestly to general reasoning tasks, but the transfer is weaker and more variable than the in-domain gains.
Harder benchmarks benefit more: on AIME24 (the most challenging math benchmark), improvements are consistently the largest percentage points (+4.5%, +7.5%, +10.6%, +21.4% across the four model sizes), while on MMLU-Pro (a broad multi-task benchmark), gains are more modest (+2.0%, +0.1%, +0.3%). This is consistent with the mechanism: compositional prompts teach structured multi-step reasoning, which is precisely what hard competition math requires but which general knowledge questions may not emphasize.

Curriculum Composition-RL: Progressive Depth (Section 4.3, Table 1 and Figure 1, Bottom-Left)

Headline result: A curriculum that progresses from Depth 1 (original MATH12K) → Depth 2 (MATH-Composition-199K) → Depth 3 (further composed prompts) on Qwen3-4B-Base yields cumulative improvements over direct Depth 2 training, with the final Depth 3 checkpoint reaching 37.9% on AIME24, outperforming several 8B baselines trained on larger datasets.

Curriculum progression (Table 1, rows "Depth 1 + 2" and "+ Depth 3" for Qwen3-4B-Base):

Depth 1 (original MATH12K): Math overall 23.8%, multi-task 50.1%, overall average 34.1% (same as the baseline row).
Depth 1 → Depth 2 (curriculum from original to compositional): Math overall improves to 31.0% (+7.2% over Depth 1 alone, +3.6% over direct Depth 2 training on MATH-Composition-199K), with AIME24 reaching 34.0% (+14.1% over Depth 1, +9.6% over direct Depth 2). Multi-task improves to 54.0% (+3.9% over Depth 1, +1.2% over direct Depth 2). Overall average reaches 40.4% (+6.3% over Depth 1, +3.0% over direct Depth 2).
Depth 1 → Depth 2 → Depth 3: Math overall reaches 32.4% (+1.4% over Depth 1 → Depth 2), with AIME24 reaching 37.9% (+3.9% over Depth 2) and AIME25 reaching 27.4% (+3.3% over Depth 2). Multi-task reaches 54.4% (+0.4% over Depth 2). Overall average reaches 42.4% (+2.0% over Depth 2). The third depth transition yields diminishing but still positive returns.

Comparison to 8B baselines (Figure 1, bottom-left, and Section 4.3 text):

The final Composition-RL 4B model (Depth 1 → 2 → 3) achieves 37.9% on AIME24, which the paper compares against published numbers from prior work using Qwen3-8B-Base on DAPO-MATH-17K:

Beyond-80/20-8B (Wang et al., 2025): 34.6%
Alpha-RL-8B (Cai et al., 2025): 28.3%
RL-ZVP-8B (Le et al., 2025): 24.6%

The paper emphasizes this as an "unfavorable" comparison (4B vs. 8B base, MATH12K vs. DAPO-MATH-17K) to underscore the effectiveness of compositional training. However, the baselines and Composition-RL differ in model scale, training data, RL algorithm details, and training duration, making this comparison more illustrative than controlled. The 37.9% figure for the 4B model is an absolute number; relative to the Depth-1 baseline on the same model (19.9% on AIME24), the curriculum provides an 18.0 percentage-point improvement.

Validation curves (Figure 1, bottom-left):

The pass@1 on AIME24 (the validation metric shown) increases steadily during Depth-1 training, plateaus, jumps when Depth-2 training begins (as the solve_all rate drops and informative gradients resume), plateaus again, and jumps again at the Depth-3 transition. The shape of the curve provides visual evidence for the paper's claim that compositional depth functions as a curriculum — each transition re-activates learning by making the training prompts harder relative to the model's current capability.

Cross-Domain Composition: Physics-MATH-Composition-141K (Section 4.4, Table 2 and Figure 1, Bottom-Right)

Headline result: RL training on Physics-MATH-Composition-141K (composed physics + math prompts) substantially outperforms both Mix Training (concatenated physics + math prompts) and Math-then-Physics (sequential domain training), achieving +5.8% and +7.1% improvements on AIME24 over Mix Training and Math-then-Physics respectively, and +2.1% and +1.3% on MMLU-Pro.

Detailed comparison (Table 2, Qwen3-4B-Base):

MATH12K-only baseline: Math overall 16.6% (AIME24: 23.3%, AIME25: 19.5%), multi-task 51.2% (GPQA: 43.7%, MMLU-Pro: 58.6%), overall average 28.1%.
Mix Training (MATH12K + Physics): Math overall drops to 14.1% (−2.5%), with AIME24 falling to 19.7% (−3.6%) and AIME25 to 16.5% (−3.0%). Multi-task improves slightly to 52.0% (+0.8%). Overall average drops to 26.8% (−1.3%). The math degradation from mixing physics prompts suggests interference — the model's math reasoning capability is diluted by the physics data when both domains compete in the same training batches.
Math-then-Physics: Math overall improves to 17.7% (+1.1% over MATH12K-only), with AIME24 reaching 25.3% (+2.0%) and AIME25 reaching 22.3% (+2.8%). Multi-task improves to 53.3% (+2.1%). Overall average reaches 29.5% (+1.4%). Sequential training avoids the interference observed in Mix Training and provides modest gains across both math and multi-task.
Physics-MATH-Composition-141K (Composition-RL): Math overall reaches 21.6% (+5.0% over MATH12K-only, +7.5% over Mix Training, +3.9% over Math-then-Physics). AIME24 reaches 32.4% (+9.1% over MATH12K-only, +12.7% over Mix Training, +7.1% over Math-then-Physics). AIME25 reaches 25.5% (+6.0% over MATH12K-only). Multi-task reaches 54.7% (+3.5% over MATH12K-only, +2.7% over Mix Training, +1.4% over Math-then-Physics). MMLU-Pro reaches 62.7% (+4.1% over MATH12K-only). Overall average reaches 32.6% (+4.5% over MATH12K-only, +5.8% over Mix Training, +3.1% over Math-then-Physics).

Subject-level breakdown (Figure 1, bottom-right):

The paper provides MMLU-Pro subject-level results for five subjects with the largest sample sizes, comparing four conditions: Math-Only (MATH12K), Math-then-Physics, Mix Training, and Physics-MATH-Composition-141K. The composition condition yields the highest accuracy on all five subjects (math, physics, law, engineering, chemistry). Notably, the math-composition condition (MATH-Composition-199K) also generalizes beyond math, improving on physics, law, and engineering relative to math-only training, without ever seeing physics data. This suggests that compositional reasoning skills themselves transfer across domains, even when the compositional data is math-only.

Key pattern: Cross-domain composition (physics → math) achieves both higher math performance and higher multi-task performance than any baseline, whereas Mix Training sacrifices math for multi-task gains, and sequential training provides only modest benefits to both. The compositional approach simultaneously improves in-domain (math) and out-of-domain (multi-task, including physics-related subjects) performance, a pattern neither baseline achieves.

Meta-Experiments: The solve_all Problem and Composition (Section 3.2, Figure 2)

Headline result: During standard RLVR on MATH12K, the solve_all ratio rises to ~75% by step 250, shrinking the effective training set to approximately 3K prompts. SPC composition reduces the solve_all rate on pre-trained reasoning models from 81.5–88.5% to 41.4–60.0%, demonstrating that composition re-activates stale prompts.

solve_all ratio curve (Figure 2, Left):

At step 0 (before any RL training), the solve_all ratio is near zero.
Within the first 50 training steps, the ratio rapidly climbs to over 50%.
By step 250 (the end of the plotted training run), the ratio stabilizes around 75%.
The solve_none ratio (not directly plotted but stated in the text) remains around 5% at step 250.

Composition effect on pre-trained models (Figure 2, Right):

On a 200-question subset of MATH500 and corresponding Depth-2 compositional prompts (~400 prompts):

OpenMath-Reasoning-1.5B: avg@8 drops from the original set to the compositional set by 19.7 percentage points; solve_all rate drops from 81.5% to 41.4%.
JustRL-1.5B: avg@8 drops by 15.4 percentage points; solve_all rate drops from 88.5% to 60.0%.
JustRL-1.5B (obtained by RL-training OpenMath-Reasoning-1.5B on normal prompts) shows improved performance on both original (+2.3%) and compositional (+6.6%) test sets, suggesting that RL on normal prompts partially transfers to compositional reasoning. This motivates the central question: does RL on compositional prompts improve normal reasoning? (The answer, per Table 1, is yes.)

Ablation Studies and Robustness Checks

Candidate set construction for $\mathcal{D}_1$ and $\mathcal{D}_2$ (Section 5.1, Table 3): The default Composition-RL configuration ( $|\mathcal{D}_1| = 20$ randomly sampled, $\mathcal{D}_2 = \mathcal{D}$ full set) is compared against two variants: (A) both $\mathcal{D}_1$ and $\mathcal{D}_2$ as small random subsets ( $|\mathcal{D}_1| = |\mathcal{D}_2| = 500$ ), and (B) $\mathcal{D}_1$ as the full set and $\mathcal{D}_2$ as a small subset ( $|\mathcal{D}_1| = 12,000$ , $|\mathcal{D}_2| = 20$ ). The default configuration achieves the best overall performance (19.1% overall average) vs. Variant A (15.7%) and Variant B (16.9%), confirming that using the full dataset for $\mathcal{D}_2$ (which determines the final answer) is critical for answer diversity, while a small pool for $\mathcal{D}_1$ (which determines the intermediate variable) is sufficient. Variant A performs comparably to the baseline (RL on original $\mathcal{D}$ , 15.7% overall average), which is notable because it uses only 1,000 distinct seed prompts to generate compositional data — suggesting that composition can partially compensate for limited data quantity.
Compositional depth curriculum vs. direct training (Table 1, Qwen3-4B rows): Direct training on Depth-2 compositional data (MATH-Composition-199K) yields an overall average of 37.4%, while the curriculum Depth 1 → Depth 2 yields 40.4% (+3.0%). This validates the curriculum approach: training on original prompts first, then transitioning to compositional prompts, outperforms starting from compositional prompts directly. The paper does not ablate the transition timing or provide a direct comparison of Depth 1 → Depth 2 vs. Depth 2 → Depth 1 (reverse order), which would have tested whether the original-then-compositional order is necessary or merely helpful.
Depth 3 composition (Table 1, Qwen3-4B, "+ Depth 3" row): Adding a third depth stage continues to improve performance over Depth 2 alone (+2.0% overall average), but the marginal gain is smaller than the Depth 1 → Depth 2 transition (+6.3%). The paper does not ablate whether training directly on Depth-3 data (without the curriculum) would be effective or whether the model would produce only solve_none responses.
Cross-domain composition vs. baselines (Table 2): The comparison includes Mix Training and Math-then-Physics as the two natural alternatives. Mix Training degrades math performance relative to MATH12K-only (16.6% → 14.1% math overall), while Math-then-Physics provides modest improvements (16.6% → 17.7%). Physics-MATH-Composition-141K substantially outperforms both. The paper does not ablate the direction of composition (math → physics vs. physics → math) or test whether the reverse composition would yield different benefits.
Verification pipeline filtering yield (Appendix D.1 and B.2): The three-step self-verification procedure removes approximately 17% of candidate compositions (240K theoretical → ~199K final), with the majority of losses at the first verification step (modification of $q_1$ ). The final error rate is cited as below 2% (from Xiao and Zhao, 2025). The paper does not ablate the verification pipeline — e.g., training on unfiltered compositions to measure the impact of erroneous prompts — which would have quantified the importance of data quality in this setting.
solve_all ratio on compositional data (Figure 2, Right): The reduction in solve_all rate from 81.5–88.5% to 41.4–60.0% is shown for pre-trained models (OpenMath-Reasoning-1.5B and JustRL-1.5B), not for the policy models during RL training. The paper does not report the solve_all ratio on compositional data during the actual Composition-RL training runs, which would more directly confirm the mechanism — i.e., that training on compositional prompts keeps the solve_all rate lower for longer compared to training on original prompts.
Scaling of gains with model size (Table 1): The monotonic increase in improvement from 4B (+3.3%) to 30B-A3B (+10.5%) is observed but not explicitly ablated — the paper does not train intermediate sizes or vary training duration to determine whether larger models benefit more per training step or simply have more capacity to absorb the compositional signal. The MoE architecture (30B-A3B) shows the largest gains, but whether this is due to model scale, architecture, or the lower baseline performance cannot be disentangled from the reported experiments.

Critical Assessment

Claim 1: "Composition-RL consistently improves reasoning capability over RL trained on the original dataset."

Supported by Table 1 across four model scales and six benchmarks. However, "consistently" must be qualified: the gains are consistent in direction (all models, all benchmarks show positive deltas), but the magnitude varies substantially — from +0.1% on MMLU-Pro for Qwen3-8B to +21.4% on AIME24 for Qwen3-30B-A3B. The largest gains appear on the hardest math benchmarks (AIME24, BeyondAIME) and the smallest on broad multi-task benchmarks (MMLU-Pro). This pattern is consistent with the mechanism (composition teaches multi-step reasoning chains, which directly transfers to competition math) but means the method is strongest where it is most needed (hard reasoning) and weaker where the connection to the training task is looser (general knowledge).

The claim is demonstrated for Qwen3 models on MATH12K. Whether the same gains would appear with different base models (e.g., Llama, DeepSeek), different training datasets (e.g., larger or more diverse math sets), or different RL algorithms (e.g., PPO, REINFORCE with different advantage estimators) is not tested. This is a single-model-family, single-dataset evaluation, albeit with four model sizes that provide some evidence of scaling consistency.

Claim 2: "Performance can be further boosted with a curriculum variant that gradually increases compositional depth."

Supported by the Depth 1 → Depth 2 → Depth 3 rows in Table 1 for Qwen3-4B, with cumulative improvements over direct Depth-2 training (+3.0% overall average for Depth 1 → 2 vs. direct Depth 2, +2.0% additional for Depth 3). The evidence is from a single run with a single model size. The transition timing is heuristic (saturation of validation performance), not automated, and the paper does not report sensitivity to transition timing or whether training on Depth-2 data from the start with more steps would eventually match the curriculum.

The comparison to 8B baselines (37.9% vs. 24.6–34.6%) conflates model scale, training data, and RL algorithm, making it a rhetorical comparison rather than a controlled experiment. The claim that a 4B model "surpasses several 8B baselines" is true in the narrow sense of reported numbers, but the baselines differ along multiple axes and were not reproduced under identical conditions.

Claim 3: "Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains."

Supported by Table 2, where Physics-MATH-Composition-141K outperforms Mix Training and Math-then-Physics. The key evidence is that composition is qualitatively different from mixing or sequential training — it improves both math and multi-task simultaneously, whereas Mix Training sacrifices math for multi-task gains. The claim is demonstrated for one specific cross-domain pair (physics → math) with one model size (4B) and two baselines. Missing controls include: (a) composing math with math at equivalent data scale (to determine how much of the gain is from composition per se vs. cross-domain composition specifically), (b) the reverse composition direction (math → physics), and (c) composition of non-STEM domains (e.g., law and history) to test generality.

Genuine weaknesses in the experimental design:

No statistical uncertainty quantification. All results are reported as point estimates from single training runs. With evaluation benchmarks as small as AIME24 (30 problems) and AIME25 (30 problems), the sampling variance of avg@32 can be substantial. Without confidence intervals, it is impossible to determine whether a +0.7% gain on MMLU-Pro (as seen for the 14B model) is meaningful or within noise. The paper would be strengthened by reporting standard errors or running multiple seeds.
Single training dataset (MATH12K). While this is a standard benchmark, the method's effectiveness is demonstrated only on a dataset of math competition problems. Whether Composition-RL would work for code generation (where chaining sub-problems is natural), science reasoning (where the paper provides only an initial cross-domain experiment), or non-STEM domains is unknown. The cross-domain results in Table 2 are a promising step but remain within STEM.
The SPC pipeline uses a 32B model (Qwen2.5-32B-Instruct) as the composition engine, which is larger than any model being trained. This raises a practical question: can Composition-RL work if the largest available model is used as the composition engine, or does the method require a model that is substantially stronger than the policy being trained? The paper does not ablate the composition model scale or quality.
No direct evidence that solve_all prompts are reactivated during Composition-RL training. Figure 2 (Right) shows that composed prompts have a lower solve_all rate for pre-trained models, but the paper does not plot the solve_all ratio during Composition-RL training (as it does for standard training in Figure 2, Left). This is a missing measurement that would directly validate the claimed mechanism. The improved performance is consistent with the mechanism, but alternative explanations — e.g., compositional data simply provides more diverse training examples, or the structured dependency has benefits independent of the solve_all reduction — cannot be ruled out.
The cross-domain composition benefits could be partially explained by data scale. Physics-MATH-Composition-141K contains ~141K prompts, while the baseline MATH12K has 12K. The Math-then-Physics baseline trains sequentially, but the total number of gradient updates is kept constant, so it sees fewer physics prompts than the compositional model. A fairer comparison might give the sequential model the same number of physics training steps as the compositional model, controlling for data quantity.
The ablation on candidate sets (Table 3) does not include a no-composition baseline with equivalent compute. All variants train on compositional data. A baseline that trains on original MATH12K for the same number of gradient updates as Variant A or B is not separately reported (the "Baseline" row in Table 3 is the original 12K training, not matched on steps). This makes it unclear whether the compositional variants underperform relative to the default configuration or also underperform relative to simply training longer on original data.
The claim of "implicit process supervision" (Section 5.2) rests on Figure 3 (Right), which shows that the intermediate variable $v_1$ is recovered with increasing accuracy during training. This is a correlation consistent with the mechanism, but it is not a causal demonstration — the improvement in $v_1$ recovery could be a side effect of general reasoning improvement rather than a direct result of compositional training. A control (e.g., training on non-composed prompts of equivalent difficulty and measuring whether $v_1$ -like intermediate reasoning also improves) would strengthen the causal interpretation.

Missing experiments that would strengthen the paper:

Ablation of the composition structure: compare SPC (sequential, asymmetric, variable-introduction) against a simpler baseline that simply concatenates two problems and asks for both answers. This would isolate whether the structured dependency (chaining) is necessary or whether merely seeing harder/longer prompts is sufficient.
Control for prompt length: composed prompts are longer than original prompts. Training on longer prompts may have independent effects on the model's ability to handle extended reasoning chains. A baseline that pads or extends prompts without adding compositional structure would isolate the length confound.
Training with different RL algorithms: testing Composition-RL with PPO (not just GRPO) would demonstrate whether the gains are specific to the group-based advantage normalization (where solve_all prompts have zero advantage) or general to any RLVR setup.
Longer training on original data with the same total compute: the paper matches gradient updates between Composition-RL and baseline. But if compositional prompts are longer (more output tokens), each gradient update for Composition-RL may consume more computation. A FLOPs-matched comparison would address whether the gains persist when accounting for the increased sequence length cost.
Evaluation on compositional test prompts of varying depth: the paper reports only standard (Depth-1) benchmarks for main results. Testing the trained models on Depth-2 and Depth-3 held-out compositional test sets would directly measure whether compositional training improves compositional reasoning ability, supporting the "compositional generalization" interpretation in Section 5.2. The meta-experiment (Figure 2, Right) evaluates this for pre-trained models but not for Composition-RL-trained checkpoints.

Summary of evidential status:

The central claim — that composing training prompts improves RLVR performance — is robustly supported across model scales and benchmarks. The more specific claims — that the mechanism operates through implicit process supervision, that the curriculum is optimal, and that cross-domain composition is uniquely effective — are supported by suggestive evidence but lack the controlled ablations that would establish causal mechanisms. The paper demonstrates that Composition-RL works (reliably, across scales), provides plausible mechanisms for why it works, but stops short of experimentally isolating those mechanisms. This is typical for a methods paper introducing a new data augmentation technique, and the evidence standard is appropriate for the contribution type (a new training data pipeline with strong empirical results), but the mechanistic claims should be treated as hypotheses consistent with the data rather than as experimentally verified.

6. Limitations and Trade-offs

6.1 The SPC Pipeline Depends on a Larger, More Capable Model Than the One Being Trained

The assumption or constraint. The entire Composition-RL data pipeline — including the three-step SPC procedure and the three-stage verification — is automated using Qwen2.5-32B-Instruct, a 32B-parameter instruction-tuned model. Every composed prompt in MATH-Composition-199K and Physics-MATH-Composition-141K passes through this model to extract variables, rewrite problems, generate connecting constraints, and self-verify the results. The paper states this explicitly in Appendix D.2: "We use Qwen2.5-32B-Instruct (Team, 2024) with step-specific prompts to implement each stage."

This is significant because the composition model is substantially larger than any policy model being trained. The experiments train Qwen3 models ranging from 4B to 30B parameters. The 32B composition model is larger than the 4B, 8B, and 14B policy models, and comparable to the 30B-A3B model (though the MoE architecture makes direct comparison difficult). This is not a neutral implementation detail — it means the method, as demonstrated, requires access to a model that is at least as capable as the largest model being trained, and likely more capable in absolute reasoning terms given that it is instruction-tuned and the policy models are base models.

The consequence. A practitioner who wants to apply Composition-RL to train a model of size $N$ must first have access to a composition model of size ≥ $N$ that is capable enough to reliably extract variables, rewrite math problems, generate correct arithmetic constraints, and self-verify. If the best available model is the one being trained, Composition-RL cannot bootstrap — you cannot use a 4B model to compose prompts for training a 4B model because the composition quality would degrade (more errors in variable extraction, more incorrect constraints, more compositions failing verification, and more erroneous prompts slipping through). The paper does not evaluate composition quality as a function of the composition model's capability, so the minimum model scale or capability threshold required for reliable SPC is unknown.

This creates a circular dependency for the most practically important use case: using Composition-RL to improve a modestly-sized model (the scenario where training data is scarcest relative to model capacity). If you only have a 7B model, can you use that same 7B model to compose prompts? Can you use a 3B model? The paper provides no evidence. The 32B composition model is a reasonable practical choice (it is publicly available), but if your organization's largest available model is, say, 14B, can you still run Composition-RL on a 30B-A3B model? If you are training a 70B model and your largest composition model is 32B, does composition quality degrade? These are open questions.

What evidence exists in the paper. The paper reports the filtering yield of the SPC pipeline (~17% of candidate compositions are rejected by the three-stage verification, Section B.2, Appendix D.1), which is a measure of the 32B model's self-consistency. It also cites Xiao and Zhao (2025) for the claim that post-filtering error rates are below 2%. These numbers are for the 32B composition model specifically. No ablation varies the composition model scale or capability.

Mitigation status. The paper does not acknowledge this as a limitation, let alone address it. The choice of Qwen2.5-32B-Instruct is presented as an implementation detail, not as a methodological constraint. A natural future experiment — compositional data quality and downstream RL performance as a function of the composition model's size and capability — is not suggested.

6.2 Compositional Prompts Are Longer, Incurring Unaccounted Training Cost

The assumption or constraint. The paper matches baseline RL and Composition-RL on number of gradient updates (Section 4.1: "we train Composition-RL for the same number of gradient updates as the baseline"). The implicit assumption is that one gradient update on a compositional prompt costs the same as one gradient update on an original prompt. This is unlikely to be true because compositional prompts are structurally longer: they concatenate two or more original problem statements plus connecting text, and they require longer responses (the model must solve multiple sub-problems sequentially rather than one standalone problem). Longer sequences mean more forward-pass tokens and more backward-pass computation per training step.

The consequence. The headline efficiency gains (e.g., Composition-RL outperforming baseline RL by +3.3% to +10.5% overall, Table 1) are stated in terms of matched gradient updates, but the total FLOPs or wall-clock time spent training is likely higher for Composition-RL because each gradient update processes longer sequences. If compositional prompts are, on average, 2× longer in input tokens plus 2× longer in output tokens compared to original prompts, then a matched "number of gradient steps" comparison actually gives Composition-RL roughly 4× more training compute. The reported gains would then be partially attributable to having more compute, not just to having better data. The paper provides no accounting of sequence length differences, total tokens processed, or FLOPs consumed, making it impossible to determine how much of the improvement is from the compositional data structure vs. from effectively training longer.

This matters acutely for practitioners with fixed compute budgets. If Composition-RL requires 3× more FLOPs per update to achieve a +5% accuracy gain, and the same FLOPs could alternatively be spent on 3× more gradient updates with the original data, the fair comparison is not the one the paper reports. The paper does not rule out the possibility that simply training longer on original data — until the total tokens consumed matches the compositional training run — would close part or all of the gap.

What evidence exists in the paper. The paper provides no measurements of: (a) average input token length for original vs. compositional prompts, (b) average output token length for responses to original vs. compositional prompts, (c) total FLOPs or GPU-hours per training run for any experiment. The maximum output length is set to 16K tokens for both original and compositional training (Section B.1), but the actual generated lengths are not reported. The training configuration (batch size 256, 8 rollouts per prompt, same number of gradient updates) is described in detail, but sequence length — which directly affects compute cost — is never quantified.

Mitigation status. Not addressed. The paper does not mention training cost as a trade-off, does not report sequence lengths, and does not attempt a FLOPs-matched comparison. This is a significant omission given that the method's core claim is about efficiency (getting more out of limited data), and efficiency is meaningless without accounting for the cost of what you are doing differently.

6.3 The Method Is Demonstrated Only on STEM Reasoning with a Single Dataset and Model Family

The assumption or constraint. All main experiments use the MATH12K training dataset (with the cross-domain extension adding the physics subset of MegaScience). All policy models are from the Qwen3 family (4B, 8B, 14B, 30B-A3B). All evaluation benchmarks are either math reasoning (AIME24, AIME25, BeyondAIME, IMOBench) or multi-task STEM/general reasoning (GPQA-Diamond, MMLU-Pro). The composition model is Qwen2.5-32B-Instruct. This is a single-model-family, single-dataset-domain evaluation, albeit with model scale variation.

The paper positions Composition-RL as a general framework for RLVR training data augmentation (Section 3.3: "Composition-RL then optimizes the RLVR objective over compositional prompts," with no domain-specific constraints in the formulation). But the method's key mechanism — extracting a numeric value from one problem's answer and injecting it as a variable into another problem — relies on the fact that math (and physics) problems have clean numeric answers that can be abstracted into variables. Whether the same SPC procedure would work for code generation (where answers are programs, not numbers), for factual QA (where answers are text strings), or for reasoning tasks without numeric outputs (logical deduction, ethical reasoning) is entirely untested.

The consequence. The paper's claims of general effectiveness ("consistently improves reasoning capability," Section 1; "highlighting the potential for cross-domain integration," Section 6) outrun the evidence, which is confined to STEM reasoning with numeric answers. A practitioner working on RLVR for code generation, dialogue, or legal reasoning cannot infer from the paper whether Composition-RL would work, how to adapt SPC for non-numeric answer types, or whether the implicit process supervision mechanism (which depends on error propagation through a numeric computation chain) would transfer. The method may be fundamentally tied to domains where problems produce intermediate values that can be abstracted and recombined — a restriction the paper does not explicitly acknowledge.

Even within STEM, the paper tests only one cross-domain pair (physics → math), only in one direction, and only at one model scale (4B). The claim that "Composition-RL enables more effective cross-domain RL" (Section 1) generalizes from a single data point.

What evidence exists in the paper. The positive evidence is all from MATH12K + MegaScience physics on Qwen3 models. The paper does not evaluate on code generation benchmarks, non-STEM reasoning tasks, or with non-Qwen model families (e.g., Llama, DeepSeek). The cross-domain experiment (Section 4.4) is limited to physics → math. The paper's claimed OOD improvements (GPQA-Diamond and MMLU-Pro in Table 1) are modest (+0.7% to +2.9% for multi-task averages) and could reflect general reasoning transfer from math, not compositional training specifically.

Mitigation status. The paper does not acknowledge this as a limitation. Section 7 lists future directions including "Expanding composition to cover more domains," which implicitly acknowledges the current scope limitation but does not characterize it as a generalizability concern. No caveats about the numeric-answer requirement or the single-model-family evaluation are mentioned.

6.4 The Implicit Process Supervision Mechanism Is Correlational, Not Causally Established

The assumption or constraint. A central explanatory claim of the paper is that compositional prompts provide "implicit process supervision" (Section 5.2): the structured dependency in composed prompts forces the model to execute intermediate steps correctly because errors in the first sub-problem propagate to a wrong final answer, and the outcome reward implicitly penalizes incorrect intermediate reasoning. The evidence for this is Figure 3 (Right), which shows that the model's accuracy at recovering the intermediate variable $v_1$ improves over the course of Composition-RL training.

The consequence. The improvement in $v_1$ recovery is consistent with the implicit process supervision hypothesis, but it is equally consistent with an alternative explanation: the model is simply getting better at math generally (the same trend would appear if you trained on any effective math dataset and measured $v_1$ recovery as a proxy for math capability). The paper provides no control that would distinguish "implicit process supervision from the composed structure" from "general reasoning improvement from more training." A natural control — measuring $v_1$ recovery on a model trained on non-composed prompts of equivalent difficulty for equivalent steps, and comparing the rate of improvement — is not provided. Without this, the claim that composed prompts cause better intermediate reasoning (as opposed to better intermediate reasoning being a correlated side effect of better math capability) is unsubstantiated.

This matters because the "implicit process supervision" framing is used to explain why Composition-RL works (Section 5.2, "Why Composition-RL Works"). If the mechanism is not uniquely attributable to composition, the design implications are different. If composition merely provides harder training examples and the improvement is from the difficulty increase rather than from the structured dependency, then other difficulty-escalation methods (harder problems, longer problems, more complex single problems) might achieve the same gains without the SPC pipeline's cost. The paper's contribution would then be a specific difficulty-escalation technique, not a qualitatively different training signal.

Moreover, the practical takeaway — that you get "process supervision for free" without training a separate verifier — is only valid if the process supervision is genuine (i.e., the gradient signals specifically penalize intermediate reasoning errors in a way that standard outcome supervision on standalone problems does not). If the benefit is merely from harder data, the "process supervision" label is misleading.

What evidence exists in the paper. Figure 3 (Right) shows one curve: $v_1$ recovery accuracy over training steps for a Composition-RL run. No control curve from a baseline training run is plotted. The paper does not measure $v_1$ recovery on the baseline model (standard RL on MATH12K) at equivalent training steps, nor does it compare the rate of $v_1$ improvement per unit of overall math improvement. The compositional generalization analysis (Figure 3, Left) shows that Composition-RL models improve on Depth-2 test prompts, but this is a measure of compositional reasoning ability, not specifically of intermediate variable recovery.

Mitigation status. Not addressed. The paper presents the $v_1$ recovery curve as confirmatory evidence without acknowledging the missing control. The "implicit process supervision" framing is stated in the abstract and section headers as an established finding, not as a hypothesis consistent with the data. A causal ablation — e.g., comparing a model trained on composed prompts against a model trained on equivalently difficult non-composed prompts, and measuring whether the composed model shows disproportionately better intermediate reasoning — is absent both from the experiments and from the suggested future work.

6.5 The Curriculum Transition Is Heuristic, Not Automated, and the Depth Scaling Does Not Demonstrate an Asymptote

The assumption or constraint. The curriculum variant of Composition-RL (Depth 1 → Depth 2 → Depth 3) transitions between compositional depths when performance "saturates" (Section 4.3: "once performance saturates, we switch to Composition-RL with Depth 2"). The saturation criterion is heuristic — the authors observe the validation curve (Figure 1, bottom-left) and manually choose transition points — not automated by a formal stopping rule, statistical test, or difficulty-estimation trigger. The paper provides no sensitivity analysis: would earlier or later transitions change the results? Is there an optimal transition point, or is any point after plateau equally good?

The consequence. A practitioner trying to replicate the curriculum faces an ambiguous decision: when exactly do you switch? The validation curve is noisy (AIME24 has only 30 problems, making pass@1 estimates high-variance), and the plateau may not be obvious in real time. If the practitioner transitions too early, the model may not have fully absorbed Depth-1 skills, making Depth-2 compositions too hard (solve_none). If they transition too late, they waste compute on saturated Depth-1 training. The paper provides no guidance on transition criteria, no sensitivity analysis, and no evidence that the reported results are robust to transition timing. This makes the curriculum variant harder to operationalize than the paper suggests.

Furthermore, the paper does not demonstrate that Depth-2 → Depth-3 provides asymptotic benefit, or that Depth 4 would provide further gains. The marginal improvement from Depth 2 to Depth 3 is +2.0% overall average (Table 1, Qwen3-4B rows), which is positive but substantially smaller than the Depth 1 → Depth 2 gain (+6.3%). The paper does not explore whether this diminishing return continues, whether there is a depth at which compositions become too long or too difficult to provide any benefit, or whether the model would eventually saturate on Depth 3 as it did on Depth 1. Without these measurements, the curriculum approach is demonstrated to work for three steps but the scaling properties are unknown.

What evidence exists in the paper. Figure 1 (bottom-left) shows the validation pass@1 curve with vertical dashed lines indicating depth transitions. The curve visually plateaus before each transition, but no quantitative criterion is stated. Table 1 reports the cumulative performance at each depth for Qwen3-4B only — no other model size is tested with the curriculum. The paper does not report ablation on transition timing, number of steps at each depth, or any automated difficulty threshold for triggering transitions.

Mitigation status. Not addressed. The paper describes the curriculum as a natural extension of Composition-RL but does not discuss the transition heuristic as a limitation, does not provide automation or guidance, and does not test robustness. The related work on online difficulty prediction (Qu et al., 2025, cited in the paper) could potentially be adapted for automated transition decisions, but this connection is not made.

6.6 The Cross-Domain Composition Evaluation Does Not Isolate Composition from Data Scale Effects

The assumption or constraint. The cross-domain experiment in Section 4.4 compares Physics-MATH-Composition-141K (~141K compositional prompts) against two baselines: Mix Training (MATH12K + MegaScience physics, ~12K + ~8.2K = ~20.2K prompts) and Math-then-Physics (sequential training on MATH12K, then physics, same total gradient updates). The compositional dataset is roughly 7× larger than the Mix Training dataset and 10× larger than MATH12K alone. The paper notes that all conditions are trained for approximately the same number of gradient updates (Section B.2), meaning the compositional model sees far more unique prompts over the course of training than any baseline.

The consequence. The performance advantage of Physics-MATH-Composition-141K over the baselines (+7.1% on AIME24 vs. Math-then-Physics; +12.7% vs. Mix Training, Table 2) is confounded by data quantity. The compositional model sees many more distinct training examples than any baseline because the compositional dataset is much larger. The obvious question — "is the gain from cross-domain composition per se, or simply from having more training data?" — cannot be answered from the reported experiments. A proper control would either: (a) down-sample the compositional dataset to match the baseline data quantity (~20K prompts), or (b) up-sample the baseline data (e.g., by repeating prompts or generating more physics prompts) to match the compositional data quantity (~141K prompts). Neither is provided.

This confound is particularly acute because the paper's core claim is about the structural property of composed prompts (cross-domain chaining), not about data scale. If the same gains could be achieved by simply collecting more physics and math prompts and mixing them (without composition), the compositional structure is incidental, not causal. The Mix Training baseline underperforms, but it also uses only ~20K prompts — far fewer unique examples than the compositional run. The Math-then-Physics baseline also uses fewer total unique physics examples (since the physics training is a continuation, not an expansion of the dataset, and the training stops when performance saturates).

What evidence exists in the paper. Table 2 reports the baseline and compositional results with dataset sizes described in Appendix B.2. The paper does not mention or control for the data scale difference. The ablation in Section 5.1 (Table 3) studies candidate set sizes within compositional data but does not compare compositional data against non-compositional data at matched dataset sizes. The mix training baseline is described as having "approximately the same number of total gradient updates" (Section B.2), but the number of unique prompts seen per run is not equalized.

Mitigation status. Not addressed. The paper presents the cross-domain results as evidence that composition is "more effective than naively combining physics and math prompts" (Section 4.4) without acknowledging that the compositional condition also enjoys a much larger pool of unique training examples. A data-scale-matched comparison is not suggested as future work.

7. Implications and Future Directions

How This Work Changes the Landscape

Composition-RL introduces a diagnostic reframing rather than a paradigm shift. The field of RLVR has operated under an implicit assumption that the primary obstacle to longer, more effective training is the solve_none problem—prompts the model cannot solve, which produce zero-advantage gradients and stall learning. Prior work has invested heavily in addressing this: advantage shaping for solve_none prompts (RL-ZVP, NGRPO), exploration-weighted sampling (Knapsack RL, Depth-Breadth Synergy), and hint-based augmentation that transforms hard prompts into easier ones (Questa, Chen et al., 2025b). The assumption is that solve_all prompts—those the model has mastered—represent a success condition, not a failure mode.

Composition-RL empirically inverts this assumption. The meta-experiment in Figure 2 (Left) demonstrates that for a standard Qwen3-4B-Base training run on MATH12K, the solve_all rate climbs to 75% by step 250, while the solve_none rate remains at roughly 5%. This means that at the stage of training where the model is presumably still improving, three-quarters of the training data has become dead weight—not because the problems are too hard, but because they are too easy. Prior work had no name for this bottleneck, no diagnostic for measuring it, and no method specifically targeting it. Composition-RL provides all three: the solve_all ratio as a diagnostic metric, the SPC mechanism as a targeted intervention, and empirical evidence that reactivating stale easy prompts through composition yields sustained improvements (Table 1, Figure 1 bottom-left).

This reframing shifts the allocation of research attention within the RLVR community. Before this paper, the dominant questions were: "How do we make hard prompts usable?" and "How do we generate more training data?" After this paper, an equally important question becomes: "How do we keep easy prompts from becoming useless, and how do we make them harder without discarding their underlying content?" This is not incremental—it opens a new axis of RLVR data management that the field did not previously recognize. It suggests that the solve_all problem is not merely a sign of training success to be tolerated, but an active source of inefficiency that can be engineered around. The paper does not solve the solve_all problem permanently—the compositional prompts themselves eventually become solve_all at sufficient training depth, which is why the curriculum variant exists—but it provides the first systematic method for delaying data staleness by ratcheting up difficulty without collecting new data.

The work also provides a unifying explanation for several previously disconnected observations. Why does training on limited prompts eventually saturate even with algorithmic improvements (JustRL, ProRL)? Because the solve_all ratio rises regardless of what RL algorithm you use—zero-variance prompts produce zero gradients under GRPO, PPO, REINFORCE, or any advantage estimator that normalizes within a group, and dynamic sampling filters them out regardless of the policy gradient formula. Why does mixing data from multiple domains sometimes hurt performance (Mix Training degrading math in Table 2)? Because adding off-domain prompts dilutes the effective training signal without addressing the solve_all bottleneck on the in-domain prompts—you still have 75% of your math data going stale, but now the model's attention is also divided across physics problems. Why do larger models benefit more from Composition-RL (Table 1, +3.3% for 4B vs. +10.5% for 30B-A3B)? Because larger models saturate the solve_all threshold faster (they master the original prompts more quickly) and have greater capacity to absorb the more complex reasoning patterns compositional prompts demand. These are not separate phenomena—they are all manifestations of the same underlying dynamic that Composition-RL makes explicit.

The paper also changes the calculus for data curation in RLVR. Before Composition-RL, the default strategy for improving RLVR performance was to collect more prompts—larger datasets (DeepMath-103K, Big-Math), synthetic generation (Logic-RL, REASONING Gym), or adaptive environments (RLV-E). This paper demonstrates that you can achieve substantial gains (+3.3% to +10.5% overall average) from a fixed 12K-prompt dataset simply by composing it with itself. The implication is not that data collection is unnecessary, but that data transformation should be considered alongside data collection as a first-class strategy for scaling RLVR. A team with a 100K-prompt dataset that is starting to saturate can either spend months collecting and verifying another 100K prompts, or they can spend a day running SPC on their existing data and generate millions of compositional prompts. The paper provides evidence that the latter path is productive, which reshapes the cost-benefit analysis of data engineering investments.

The cross-domain results (Table 2, Figure 1 bottom-right) further suggest that composition may be more effective than multi-task training for knowledge integration, if the metric is simultaneous improvement on both source domains. Mix Training sacrifices in-domain math performance for modest multi-task gains (16.6% → 14.1% math overall); sequential training provides only modest gains on both (16.6% → 17.7% math, 51.2% → 53.3% multi-task). Cross-domain composition improves both simultaneously and substantially (16.6% → 21.6% math, 51.2% → 54.7% multi-task). This is a finding about the structure of effective multi-domain training signals: it is not enough to expose the model to physics and math separately; the model needs to practice using physics outputs as math inputs. The standard multi-task learning paradigm—which treats domains as parallel tasks—may be fundamentally missing this integrative dimension, and composition provides a mechanism for supplying it.

Finally, the paper establishes a practical lower bound on the effectiveness of process supervision without process verifiers. The Standard RLVR pipeline uses a binary outcome verifier (Math-Verify) that checks only the final answer. The standard way to improve on this is to train a process reward model (PRM) that scores intermediate steps, which is expensive, requires on-policy data, and introduces over-optimization risks. Composition-RL demonstrates that by structuring prompts to make intermediate steps causally necessary for the final answer, you can extract process-level training signals from an outcome-only verifier. The v_1 recovery curve (Figure 3, Right) shows that the model learns to execute the intermediate computation correctly, even though it is never explicitly rewarded for doing so. This does not replace PRMs—a true PRM can provide fine-grained feedback on every step, while composition provides only binary feedback on the entire chain—but it offers a zero-cost approximation that may be sufficient for many training regimes. For research teams without the resources to train and maintain a PRM, this is a directly actionable takeaway.

Follow-Up Research This Work Enables

Causal isolation of the implicit process supervision mechanism. The paper's most theoretically interesting claim—that composed prompts provide implicit process supervision (Section 5.2)—is supported only by correlational evidence (the v_1 recovery curve improving over training). A definitive test would compare two training conditions on a controlled set of problems: (a) SPC-composed prompts where the intermediate variable v_1 is causally necessary for the final answer, and (b) length-and-difficulty-matched non-composed prompts where the same intermediate computation is not causally load-bearing (e.g., a two-part problem that asks for both v_1 and the final answer, but where the final answer can be computed independently). If the implicit process supervision mechanism is genuine, condition (a) should show disproportionately better intermediate reasoning accuracy relative to overall math improvement compared to condition (b). This experiment would also test whether the benefit comes from difficulty escalation or from the causal structure specifically. A negative result (equal intermediate reasoning improvement in both conditions) would substantially weaken the paper's explanatory framework and suggest that Composition-RL is best understood as a difficulty-scaling technique, not a process supervision technique.

Compositional depth scaling with automated curriculum transitions. The paper demonstrates a manual curriculum (Depth 1 → 2 → 3) with heuristic saturation-based transitions but provides no guidance on automation, optimal timing, or asymptotic behavior. A natural follow-up would develop an online difficulty estimator that monitors the solve_all ratio on a held-out compositional probe set at each depth and triggers a depth increase when the ratio exceeds a threshold (e.g., 70%). This estimator could be trained using the approach of Qu et al. (2025), which the paper already cites for online difficulty prediction, or could simply track the fraction of compositional prompts that are solve_all in the dynamic sampling oversample batches. The experiment would measure: (a) whether automated transitions match or exceed manual transitions in final performance, (b) sensitivity to the solve_all threshold, and (c) whether the diminishing returns from Depth 2 → 3 (+2.0% overall average) continue, plateau, or reverse at Depth 4. A finding that Depth 4 provides zero or negative gains would establish a practical composition depth ceiling for MATH12K-scale data, while continued gains would suggest that composition depth is an unbounded curriculum axis.

Composition-RL on code generation with execution-based verifiers. The paper focuses on math reasoning because numeric answers are cleanly abstractable into variables. Code generation is the natural next domain: programming problems have verifiable correctness (unit tests), produce structured outputs that can be abstracted (function return values, intermediate data structures), and naturally decompose into sub-problems (implement helper function A, then use it in function B). A Composition-RL for code experiment would: (a) use SPC to compose two programming problems by introducing a variable that represents the output of the first function and is required as an input to the second function, (b) train on compositional code prompts with execution-based verification (pass/fail on test cases), and (c) evaluate on standard code benchmarks (HumanEval, MBPP, LiveCodeBench). The key research question is whether the mechanism generalizes beyond numeric abstraction: can composition work when the intermediate variable is a data structure, a string, or a function rather than a number? Positive results would dramatically expand the scope of Composition-RL; negative results would establish that the numeric-answer abstraction is a hard requirement and limit the method to STEM domains with quantitative outputs.

Data-scale-matched cross-domain composition vs. expanded single-domain baselines. The cross-domain experiment in Table 2 confounds compositional structure with data quantity (Physics-MATH-Composition-141K vs. Mix Training on ~20K prompts). A rigorous follow-up would: (a) construct a Mix Training (143K) baseline by upsampling or augmenting the physics and math datasets to match the compositional dataset size, (b) construct a Math-Only (143K) baseline by generating additional math prompts (via synthetic augmentation or SPC within math only) to match the same scale, and (c) compare all three at matched gradient steps. This would isolate the cross-domain composition effect from the data quantity effect. If Physics-MATH-Composition-141K still outperforms the scale-matched baselines, the cross-domain integration claim is robust. If the scale-matched baselines close the gap, then composition's benefit is primarily a data-scaling effect, and cross-domain chaining is incidental. This experiment is critical for distinguishing whether cross-domain composition is a genuinely new multi-domain training paradigm or simply a convenient way to generate more data when cross-domain prompts are available.

Composition-RL without a separate composition model. The paper uses Qwen2.5-32B-Instruct—a model larger than any policy being trained—as the composition engine. This raises a practical barrier: what if you don't have access to a model larger than the one you're training? A direct test would: (a) run the SPC pipeline using the policy model itself (e.g., Qwen3-4B-Base at various checkpoints during training) as the composition model, (b) measure composition quality (verification pass rate, downstream RL performance) as a function of the policy model's capability, and (c) determine whether there exists a capability threshold below which composition quality degrades to the point that compositional training no longer helps. The result would either validate that self-composition is viable (the policy model can compose its own training data, enabling a fully bootstrapped pipeline) or establish a minimum composition model scale requirement. If the latter, it would reframe Composition-RL as a technique that requires a strong teacher model—a cost that must be factored into the method's efficiency calculus.

Composition-RL with non-mathematical reasoning tasks. The paper acknowledges in Section 7 the need to expand "composition to cover more domains," but the specific challenge is that SPC relies on extracting numeric values and injecting them via arithmetic constraints. For non-numeric domains, what is the analog? Possible directions: (a) logical reasoning: compose two logical deduction problems by making the conclusion of the first problem a premise in the second, (b) factual QA: compose two questions by making the answer to the first question a required entity for the second (e.g., "Who was the president during Event X?" → "What was President Y's policy on Issue Z?"), (c) summarization or explanation: compose tasks by requiring the model to first summarize a document, then answer a question that depends on the summary. Each of these would require adapting the SPC variable-extraction mechanism to non-numeric abstractions and developing domain-specific verification pipelines. A systematic study across 3-4 non-STEM domains would establish whether Composition-RL is a general RLVR augmentation strategy or a math-specific trick. Negative results on any domain would help delineate the method's applicability boundary, which is currently undefined.

Practical Applications and Downstream Use Cases

Maximizing the ROI of existing verifiable prompt collections. Organizations that have invested in curating verifiable training prompts—whether for math (MATH, GSM8K), science (MegaScience, UGPhysics), or code (APPS, CodeContests)—can use Composition-RL to extend the useful life of those datasets without additional collection effort. The paper's results on MATH12K demonstrate a concrete template: take your existing 12K prompts, run SPC with a capable off-the-shelf model (Qwen2.5-32B-Instruct or equivalent) for a few GPU-days, and generate a 199K-prompt compositional dataset that yields +3.3% to +10.5% overall average improvement across four model scales (Table 1). The cost of generating the compositional dataset is fixed and amortized over all future training runs; the benefit scales with model size (larger models benefit more, per the scaling trend). For a team training a 30B-class reasoning model, the +10.5% overall gain from composition represents a capability improvement that would otherwise require collecting and verifying tens of thousands of new prompts—an expensive, slow process. The paper's open-source release of MATH-Composition-199K and Physics-MATH-Composition-141K, plus the codebase and SPC prompts, makes this an immediately replicable recipe.

Curriculum-based training for sustained RLVR improvement. The curriculum variant (Depth 1 → 2 → 3) demonstrates a training schedule where the model never plateaus for long: when original prompts become solve_all, switch to Depth-2 compositions; when those become solve_all, switch to Depth-3. The validation curve in Figure 1 (bottom-left) shows three distinct phases of improvement, each initiated by a compositional depth increase. For teams running long-horizon RLVR training (millions of steps, as in ProRL or DeepScaleR), this provides a principled mechanism for maintaining the informative fraction of the training data over time. The practical implementation is straightforward: pre-generate compositional datasets at depths 1 through 3 (or higher, depending on context length constraints), monitor the solve_all ratio during training, and swap datasets when the ratio exceeds a threshold. The 4B model trained with this curriculum reaches 37.9% on AIME24 (Table 1), competitive with 8B models trained on larger datasets—a concrete demonstration that sustained data utility translates to sustained capability improvement without requiring new data.

Cross-domain integration training for STEM reasoning models. For organizations building general-purpose STEM reasoning models (covering math, physics, chemistry, engineering), the cross-domain composition result in Table 2 provides a data-efficient alternative to naive multi-domain training. Mixing physics and math prompts degrades math performance (16.6% → 14.1% math overall); composing physics into math problems improves both math and multi-task simultaneously (16.6% → 21.6% math, 51.2% → 54.7% multi-task). The practical takeaway is: if you have datasets from two related STEM domains, do not simply concatenate them or train sequentially. Instead, run SPC with one domain feeding into the other, generating a compositional dataset that forces the model to integrate knowledge across domains. The Physics-MATH-Composition-141K dataset and the MMLU-Pro subject-level results (Figure 1, bottom-right, showing composition outperforming all baselines on math, physics, law, engineering, and chemistry) provide a proof of concept and a starting dataset. The approach likely extends to other STEM domain pairs (chemistry → math, engineering → physics) with the same SPC pipeline, since all involve numeric answers that can be abstracted into variables.

When to Prefer This Method

The paper does not explicitly position Composition-RL against named alternatives as a binary choice with well-defined tradeoffs. The method is presented as complementary to existing RLVR techniques—it can be combined with any RL algorithm (GRPO, PPO, REINFORCE), any verifier (rule-based, model-based), and any prior method for handling solve_none prompts (advantage shaping, exploration weighting). The curriculum variant explicitly stacks with standard RL training (start with original data, then add composition). The cross-domain variant stacks with multi-domain training (composition is an alternative to mixing or sequencing, but the paper does not claim universal superiority or specify conditions where mixing would be preferred). The paper's contribution is a new data transformation tool, not a replacement for existing algorithmic or data-collection strategies. Framing a "prefer Composition-RL when X, prefer alternative Y when Z" decision matrix would invent a tradeoff the paper does not establish.

The one clear conditional from the paper is implicit in the model scale trend: larger models benefit more from Composition-RL (+10.5% for 30B-A3B vs. +3.3% for 4B, Table 1), suggesting that the method's value increases with the capability of the model being trained. This is consistent with the mechanism—larger models saturate the original data faster and have greater capacity to exploit compositional complexity—but it is an empirical observation, not a decision rule the paper proposes.