Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

ArXiv: 2604.12002

🎯 Pitch

Even a wrong answer contains useful information. SD-ZERO shows that by training a model to revise its own incorrect responses—then distilling that revision ability back into the base generator—you can convert a single binary reward into dense, token-level supervision, dramatically boosting sample efficiency and beating standard RL with zero external data.

---

1. Executive Summary

This paper introduces Self-Distillation Zero (SD-ZERO), a two-phase post-training method that transforms sparse binary outcome rewards into dense token-level self-supervision without requiring an external teacher or high-quality demonstrations, using Qwen3-4B-Instruct and Olmo-3-7B-Instruct evaluated across eight math and code reasoning benchmarks. SD-ZERO proceeds through Self-Revision Training (SRT) — fine-tuning the model on self-generated revision traces where it conditions on its own incorrect responses and their binary rewards to produce corrected solutions — followed by on-policy self-distillation, where the SRT model acts as its own teacher, providing token-level KL-divergence targets that the generator internalizes to produce more compact and accurate answers directly. The method improves average accuracy over base models by at least 10% (reaching 60.3% for Qwen3-4B and 51.5% for Olmo-3-7B) while reducing inference-time response length by roughly 2× compared to SRT alone, establishing that a model can bootstrap from its own revision behavior to achieve gains matching or exceeding RL-based baselines like GRPO on matched training sample budgets — an effect that compounds through iterative teacher synchronization, yielding an additional 3% gain in subsequent distillation rounds.

2. Context and Motivation

The Core Problem: Sparse Rewards Make Post-Training Expensive

The fundamental problem this paper tackles is the sample inefficiency of reinforcement learning from verifiable rewards (RLVR). In domains like math and code, we can check whether a model's final answer is correct, giving us a binary reward: 1 for right, 0 for wrong. This is enormously useful because it scales — we can verify millions of answers automatically without human annotators — but it provides only a single bit of information per generated response. As the paper puts it:

"a binary reward per response provides no information about which intermediate reasoning steps were sound. This sparse supervision makes training expensive: the model must discover good reasoning by comparing and contrasting many self-generated responses."

To understand why sparsity is crippling, consider what the model must learn during an RLVR process like GRPO (Shao et al., 2024a). It generates, say, an 8-step chain of reasoning about a geometry problem. The verifier says "wrong." The model receives no signal about which of those 8 steps went off track. Was the initial diagram setup correct but the angle calculation flawed? Did it apply a theorem to the wrong triangle? Did it make an arithmetic slip on the last line? The binary reward treats all of these failures identically. The model must therefore generate many responses — correct and incorrect — and implicitly learn which reasoning patterns correlate with success by comparing their outcomes. This is computationally expensive: GRPO, as configured in this paper, requires generating 4–8 responses per training question to get a meaningful training signal from group-relative advantage comparisons.

This sample inefficiency isn't merely an academic concern. It directly translates to higher training costs, longer iteration cycles for practitioners fine-tuning models on new domains, and barriers to entry for researchers without access to large GPU clusters. If every improvement in reasoning capability requires generating tens of thousands of responses for the RL algorithm to extract usable signal from sparse rewards, the pace of progress slows dramatically. A method that could achieve the same performance with fewer generated samples — or extract richer supervision from each sample — would directly reduce the cost of developing capable reasoning models.

Why This Problem Matters Now

The paper's framing arrives at a pivotal moment in the post-training landscape. Several converging trends make the sparse-reward problem pressing:

RLVR has become the dominant post-training paradigm. Following DeepSeek-R1 (Guo et al., 2025), reinforcement learning with verifiable rewards has emerged as the go-to method for improving reasoning in language models. Systems like DAPO (Yu et al., 2025), GSPO (Zheng et al., 2025), and numerous industrial training pipelines now standardize around binary-reward RL as the final stage of model development. If this paradigm is fundamentally sample-inefficient, the entire field bears that cost.

On-policy distillation offers an alternative but expects too much. A parallel line of work has shown that distillation — training a student to match a teacher's token-level output distribution — can be far more sample-efficient than RL because it provides dense supervision at every token position. On-policy distillation methods (Agarwal et al., 2024; Gu et al., 2024) improve on classic distillation by having the student generate its own rollouts and receive teacher feedback on those rollouts, eliminating off-policy distribution mismatch. However, these methods assume access to an external stronger teacher model that can produce high-quality token-level supervision. This external teacher is exactly what RLVR was designed to avoid needing: RLVR works with just a binary reward, not a stronger model's reasoning traces.

Self-distillation removes the external teacher but re-introduces a data dependency. More recent work — OPSD (Zhao et al., 2026a), SDFT (Shenfeld et al., 2026a), and SDPO (Hübotter et al., 2026) — removes the external teacher by having the model condition on privileged information (such as a correct solution) to serve as its own teacher. But as the paper notes:

"such methods still require high-quality demonstrations that are much better than the model's responses. These demonstrations are typically assumed to come from an external teacher (OPSD, SDFT) or repeated generation and filtering from the model itself (SDPO). Collecting such supervision can be unavailable or prohibitively expensive to collect."

This creates a frustrating tradeoff: RLVR needs no external teacher but provides only sparse supervision, while distillation provides dense supervision but needs high-quality reference solutions. The field lacks a method that achieves both: dense token-level supervision without requiring gold reasoning traces.

Where Prior Approaches Fall Short

The paper identifies specific weaknesses in each competing paradigm that motivation the search for a third approach.

RLVR: Sparse reward, sample-hungry, unclear gains beyond distribution sharpening. While GRPO and related methods have demonstrated impressive results, they come with documented limitations. The paper cites Yue et al. (2025), who "suggest that RLVR-like methods mainly sharpen the output distribution," meaning the model becomes more confident in correct answers it could already produce but may not genuinely expand its reasoning capabilities — it simply learns to select better outputs from its existing distribution rather than generating qualitatively better reasoning paths. This distribution-sharpening interpretation would explain why GRPO in this paper's experiments (Table 1) achieves only modest gains on harder benchmarks like AMOBench (11.0% for Qwen3-4B) compared to SD-ZERO's 16.0%.

Furthermore, GRPO's requirement for groups of responses per question — typically 4–8 — means that for every question in the training set, the model must generate multiple answers, consume tokens for each, compute advantages, and update. This group-structure requirement both increases the total generation budget and constrains the training setup to maintain these groups. The paper's sampling budget analysis in Appendix C.2 shows that GRPO and RFT both use approximately 60K total generations for 15K training questions, while SD-ZERO achieves stronger results with 49K total generations across its two phases.

RFT: Discards the most valuable data — the failures. Rejection Fine-Tuning (Yuan et al., 2023) is the simplest approach: generate multiple responses, keep only the correct ones, and fine-tune on those. The obvious problem is that incorrect responses are thrown away. But as the paper points out, those incorrect responses contain precisely the information the model most needs:

"Unlike RFT, which removes incorrect reasoning entirely, SRT preserves the failed attempt as context, allowing the model to learn from its own mistakes."

This matters practically: on harder benchmarks like AMOBench, RFT shows "minimal gains" (the paper's words in Section 3.2), and on HMMT25, RFT achieves 37.1% while SRT achieves 40.0% — even though SRT trains on only 6K examples vs. RFT's 15K. The lesson is that how you use the data matters at least as much as how much data you have. Seeing both the mistake and the correction teaches more than seeing only the final correct answer.

SFT on external demonstrations: Distribution shift and availability. Supervised Fine-Tuning on expert traces (like DeepSeek-R1 outputs) suffers from two problems. First, external demonstrations may not match the student model's output style or error patterns. The paper shows evidence of this in Table 1: SFT on DeepSeek-R1 responses actually degrades Qwen3-4B-Instruct on AMOBench (from 9.8 to 7.3) and LiveCodeBench (from 61.8 to 57.2). The knowledge is transferred, but the mode of expression is misaligned — the student model can't internalize reasoning patterns expressed in a form it wouldn't naturally generate. Second, external demonstrations aren't available for many domains. DeepSeek-R1's reasoning traces exist for common benchmarks but not for proprietary or niche tasks. A method that requires expert demonstrations inherits the availability constraints of those demonstrations.

Self-distillation: Privileged context ≠ self-supervision from sparse feedback. Methods like SDFT work by giving the teacher model access to a correct solution as context, then having it generate token-level targets for the student. This works well when correct solutions exist, but it doesn't transform sparse rewards — it replaces sparse rewards with pre-existing dense supervision. As the paper's ablation in Table 8 demonstrates, SDFT given only final-answer supervision (no gold solution traces) performs no better than the base model across math benchmarks (averaging 49.5% vs. 49.7% for base SDFT with full solutions). This means SDFT cannot convert binary reward signals into effective training supervision; it requires the dense signal to already exist. The paper draws a sharp contrast here: SD-ZERO is specifically designed to operate in the setting where only final-answer correctness is available.

The Missing Capability: Conditioning on One's Own Mistakes

The paper articulates a specific capability gap that existing methods don't address:

"Can the model condition its own initial attempts (possibly incorrect) and their sparse rewards, and provide improved dense supervision to itself?"

This is a precise formulation of what's missing. The model has access to three things: (1) the question, (2) its own first attempt at an answer (which may be wrong), and (3) a binary reward signaling correctness. Can those three ingredients be combined to produce token-level feedback that is more informative than the binary reward alone? Prior work provides no mechanism for doing this. RLVR methods see the binary reward but provide no token-level feedback. Distillation methods provide token-level feedback but need a source of high-quality token-level supervision (an external teacher or gold solution). Self-distillation methods use token-level feedback from a conditional model but condition on privileged information (the correct answer), not on the model's own failed attempt plus its binary reward.

The paper's key insight — and what distinguishes SD-ZERO from all prior work — is that a model fine-tuned to revise its own incorrect outputs can serve as its own teacher, and the act of conditioning on the incorrect attempt plus the binary reward transforms the sparse reward into dense token-level feedback. The reviser, having been trained on outcome-conditioned self-revision traces, "knows" how to generate a correct solution when shown an incorrect attempt labeled as incorrect. When the student produces a flawed reasoning chain, the reviser's token-level distribution — conditioned on that flawed chain and the r=0 signal — concentrates probability mass on tokens that would correct the error (the "token-level self-localization" property demonstrated in Figure 4). The student, trained to match this distribution via KL minimization, therefore receives gradients that specifically target the tokens responsible for mistakes.

This is functionally analogous to what a Process Reward Model (PRM) provides: dense step-level correctness estimates. But while PRMs require either human step-level annotations (Lightman et al., 2024) or Monte Carlo rollouts from a separate reward model (Wang et al., 2024), SD-ZERO's reviser produces this signal by conditioning the same model on its own output and a binary outcome. There is no separate reward model, no step-level data collection, and no auxiliary training objective for the verifier. The reviser simply "prompts itself" into providing richer feedback.

How This Paper Positions Itself

SD-ZERO sits at the intersection of three lines of work but synthesizes them into something none individually achieves. Table 2 in the paper provides the explicit taxonomy:

• RLVR (GRPO): on-policy ✓, dense supervision ✗, self-teacher ✗, can condition on wrong attempts ✗
• On-Policy Distillation: on-policy ✓, dense supervision ✓, self-teacher ✗, can condition on wrong attempts ✗
• Self-Distillation (OPSD/SDFT/SDPO): on-policy ✓, dense supervision ✓, self-teacher ✓, can condition on wrong attempts ✗
• SD-ZERO: on-policy ✓, dense supervision ✓, self-teacher ✓, can condition on wrong attempts ✓

The final column — the teacher's ability to condition on the student's incorrect attempt — is the novel axis. This is not a small technical tweak; it's what enables the entire pipeline to function without gold demonstrations. The reviser teacher doesn't need to know the correct answer in advance. It needs to see the wrong answer, receive the signal "this is wrong," and produce a correction. The SRT phase teaches this capability; the Self-Distillation phase leverages it.

The paper's ambition is to show that this approach is not merely competitive with existing methods under constrained budgets, but that it can serve as a better foundation for iterative self-improvement. The "iterative self-evolution" experiment in Section 3.4 (Figure 5) demonstrates that after one round of Self-Distillation, the improved model can serve as a stronger reviser teacher for a second round, yielding an additional 3% gain. This suggests that SD-ZERO's design is not a one-shot training recipe but a capability amplifier: once the model learns to self-revise, that revision capability can be recursively distilled into generation, and the improved generation can in turn produce better revisions. The paper explicitly frames this as a loop that "can continue to self-evolve through iterated teacher synchronization, requiring only the initial attempt and corresponding binary reward in the teacher's context" (Section 3.4).

This positioning matters because it reframes binary rewards from a limitation to be tolerated (as in RLVR) into a resource to be leveraged. The sparse reward isn't the problem — the lack of a mechanism to unpack it into dense signal is the problem. SD-ZERO proposes that mechanism, and the paper's experiments are designed to test whether that unpacking actually produces training signals that improve reasoning more effectively than either accepting sparsity (GRPO) or requiring external densification (SDFT with gold solutions).

3. Technical Approach

This is primarily a method-design paper that proposes a two-phase training pipeline for transforming sparse binary outcome rewards into dense token-level supervision without an external teacher, using a model conditioned on its own incorrect attempts and their binary rewards as its own reviser and teacher.

3.1 Reader Orientation

SD-ZERO is a two-phase post-training system that takes a base language model and a dataset of questions with verifiable answers (but no gold solution traces), and produces a model that generates more accurate reasoning by first learning to revise its own mistakes and then distilling that revision behavior into single-pass generation. The core problem it solves is that binary correctness rewards — which are easy to obtain at scale — give no information about where in a reasoning chain the model went wrong, making RL-based training sample-inefficient. The "shape" of the solution is a bootstrapping loop: a single model plays two roles (generator and reviser), learns to transform binary rewards into token-level correction signals by conditioning on its own failed attempts, and then internalizes that correction behavior so it produces better answers directly without needing multi-turn revision at inference time.

3.2 Big-Picture Architecture (Diagram in Words)

The SD-ZERO pipeline has five major components operating in two sequential phases:

1. Base Language Model ($\pi_\theta$) — the starting pretrained-and-instruct-tuned model (Qwen3-4B-Instruct or Olmo-3-7B-Instruct). It serves as both the initial generator of solutions and the raw material that gets fine-tuned into a reviser.

2. Binary Verifier ($r(y, a) \in \{0, 1\}$) — an external function that checks whether a generated response's extracted final answer matches the ground-truth answer. This is the only supervision signal available throughout training; no gold reasoning traces, no step-level correctness labels, no external teacher model.

3. Phase 1: Self-Revision Training (SRT) — a supervised fine-tuning stage where the base model generates initial responses, is prompted to revise incorrect ones (or rephrase correct ones) using outcome-conditioned control prompts, and is trained on filtered (correct) revision traces using a combined generation-and-revision loss. The output is the SRT model ($\pi_{\theta_{\text{SRT}}}$) which has acquired the ability to meaningfully correct its own mistakes when shown an incorrect attempt labeled as incorrect.

4. Phase 2: On-Policy Self-Distillation — a distillation stage where the SRT model is cloned into two roles: a student generator (parameters updated) and a frozen reviser teacher (parameters fixed at $\theta_{\text{SRT}}$). The student generates on-policy responses; the reviser conditions on those responses and their binary rewards to produce token-level target distributions; the student is trained via KL divergence to match those distributions. The output is the SD-ZERO model, which internalizes revision behavior into more compact, proactive generation.

5. Iterative Teacher Synchronization (optional extension) — after one epoch of Self-Distillation, the improved student can replace the frozen teacher, enabling a second round of distillation that yields further gains (Section 3.4, Figure 5).

Information flows as follows: questions enter the system → Phase 1 generates initial responses, checks correctness, constructs control prompts, generates revisions, filters for correctness, fine-tunes on revision traces → Phase 2 samples on-policy responses from the current student, checks correctness, the frozen reviser teacher conditions on the response + reward to produce next-token distributions, the student updates via KL minimization → the cycle repeats per training step, and optionally, the teacher is synchronized with the updated student for further rounds.

3.3 Roadmap for the Deep Dive

• First, the formal problem setup — what data we have, what the model is, what the binary reward is, and what we are NOT assuming (no gold solutions, no external teacher). This establishes the constraints that motivate every design choice.
• Second, Phase 1 (SRT) in full detail: how self-revision traces are collected, the structure of the outcome-conditioned control prompts, the construction of the revision dataset, the two-term loss function, and why both terms are necessary.
• Third, Phase 2 (Self-Distillation) in full detail: the student-teacher split, how the teacher conditions on the student's response and reward to produce token-level targets, the KL divergence objective, and why this is on-policy rather than off-policy.
• Fourth, the training algorithm as a unified procedure (Algorithm 1), including the data splits, sampling budgets, and the iterative teacher synchronization extension.
• Fifth, key design decisions and their justifications: why the two-phase structure is necessary, why correctness filtering matters, why the loss has two terms, why on-policy sampling matters, and what happens if you skip Phase 1.
• Sixth, connection to the token-level self-localization property (Section 4.1), explained here as a mechanism that makes the teacher's feedback informative — the reviser doesn't just penalize the whole response, it concentrates probability mass shifts on specific erroneous tokens.

3.4 Detailed, Sentence-Based Technical Breakdown

Problem Setup and Notation

The paper assumes a dataset $\mathcal{D} = \{(x_i, a_i)\}_{i=1}^N$, where $x_i$ is an input problem (e.g., a math question) and $a_i$ is its ground-truth final answer (e.g., a number or expression). Crucially, the paper does not assume access to gold solution traces — no step-by-step reasoning demonstrations are provided. The base model $\pi_\theta$ is a pretrained language model that maps an input $x$ to a distribution over reasoning responses $y$ according to $\pi_\theta(y \mid x)$. For each generated response $y$, a binary verifier extracts the final answer and checks it against $a$:

$r(y, a) = \begin{cases} 1 & \text{if the extracted answer matches } a \\ 0 & \text{otherwise} \end{cases}$

where $r(y, a)$ is the binary reward, $y$ is the model's complete reasoning response (including all intermediate steps and the final boxed answer), and $a$ is the ground-truth answer.

What it computes: a single bit of information per response — correct or incorrect — with no information about which parts of the reasoning chain contributed to the outcome. This is the sole supervision signal available throughout both training phases.

Why this form: the binary verifier is the minimal supervision that verifiable domains provide. It requires no human annotation, no step-level labeling, and no external model. The entire contribution of SD-ZERO is showing that this minimal signal can be transformed into dense token-level supervision through self-revision conditioning.

Phase 1 (SRT): Outcome-Conditioned Self-Revision Training

Phase 1 addresses a specific capability gap: base instruct models are reasonably good generators (they can produce correct answers some fraction of the time) but poor revisers — when shown their own incorrect response and told it's wrong, they typically cannot produce a corrected version that fixes the specific error. Figure 3 quantifies this: the base Qwen3-4B-Instruct model gains only 1.1 percentage points from revision (59.6% → 60.7% on AIME24), while the SRT-trained model gains 5.0 points (66.7% → 71.7%). The purpose of Phase 1 is to teach the model to revise, specifically to condition on an initial attempt and a binary reward signal and produce an improved response.

Collecting Self-Revision Traces

The SRT data collection pipeline operates on a subset of $N_1$ training examples (6K out of 15K total) and proceeds through four steps:

Step 1: Initial sampling. For each input $x$, sample one initial reasoning response from the base model:

$y_{\text{init}} \sim \pi_\theta(\cdot \mid x)$

The paper uses temperature 0.7 for all sampling during training (as specified in Section 3.1). One response per question is sampled at this stage (though the paper notes that "constructing the SRT dataset requires collecting multiple sampled responses per question in order to obtain successful revisions" — the multiplication comes from step 3, not step 1).

Step 2: Verification and control prompt construction. The binary verifier checks $y_{\text{init}}$ against the ground-truth answer $a$, producing reward $r \in \{0, 1\}$. Based on this reward, a control prompt $P_r$ is constructed:

$P_r = \begin{cases} \text{"Let me rephrase the above solution."} & \text{if } r(y_{\text{init}}, a) = 1 \\ \text{"Wait, this response is not correct, let me start over."} & \text{if } r(y_{\text{init}}, a) = 0 \end{cases}$

What this computes: a natural-language instruction that tells the model (a) whether the previous response was correct or incorrect, and (b) what action to take — rephrase if correct, start over if incorrect.

Why this form: the two distinct prompts serve different purposes. For correct responses, "rephrase" encourages the model to produce an alternative valid solution (which may be more concise or use a different approach), effectively providing positive examples of correct reasoning. For incorrect responses, the explicit error acknowledgment ("this response is not correct") combined with a restart instruction teaches the model to recognize failure and attempt correction. The paper observes that rephrasing correct responses "typically produces a shorter answer, which may additionally encourage more concise responses during the second phase" — an emergent property, not an explicit design goal. Section 2.1 states this explicitly, and it matters because conciseness becomes a key benefit of the Self-Distillation phase.

Step 3: Revision generation. The same model is prompted with the original problem, the initial response, and the control prompt to generate a revised response:

$y_{\text{revised}} \sim \pi_\theta(\cdot \mid x, y_{\text{init}}, P_r)$

The paper samples three revisions per initial response (as specified in Appendix C.1): for each correct initial response, the model generates three rephrased responses; for each incorrect initial response, three corrected responses. This multiplicity is necessary because not all revision attempts succeed — some revisions remain incorrect, and filtering (step 4) removes those.

Step 4: Filtering. Only revision traces where the revised response is correct are retained:

$\mathcal{D}_{\text{REVISION}} = \{(x, y_{\text{init}}, P_r, y_{\text{revised}}) \mid r(y_{\text{revised}}, a) = 1\}$

What this filters: revision attempts that failed to produce a correct answer. The paper reports (Appendix C.1) that starting from 10K seed questions, this pipeline produces approximately 6K filtered self-revision traces (from 5K correct + 5K incorrect initial responses, each with 3 revision attempts → 30K revision attempts → 6K successful). This means roughly 20% of revision attempts succeed, a number that varies by base model capability and problem difficulty.

Why filtering is essential: Appendix G.1 provides direct evidence that unfiltered self-revision data hurts performance. On the Countdown task with Qwen2.5-7B, unfiltered self-revision achieves pass@1 of 0.529 compared to 0.630 for filtered self-revision — a 19% relative degradation. Noisy revision traces actively teach the model incorrect reasoning. The paper notes in Appendix G (Table 15) that "noisy revision traces actively hurt" and that filtering is responsible for a 19% relative gain. This is a critical design choice: SRT is not simply "train on all revision attempts," it is "train only on successful revisions," meaning the model sees only trajectories where the revision process worked.

The Self-Revision Training Objective

The SRT model is fine-tuned on the filtered revision dataset $\mathcal{D}_{\text{REVISION}}$ using a combined loss with two terms. Let $y' = y_{\text{revised}}$ denote the correct revised response for brevity, and let $|y'|$ be its token length (this is the standard notation from the paper's Equation 1, where $y'_t$ indexes tokens of the revised response).

The revision loss trains the model to produce the correct revision when conditioned on the problem, the initial attempt, and the reward prompt:

$\mathcal{L}_{\text{revision}}(\theta) = \mathbb{E}_{(x, y_{\text{init}}, P_r, y') \sim \mathcal{D}_{\text{REVISION}}} \left[ -\sum_{t=1}^{|y'|} \log \pi_\theta\big(y'_t \mid x, y_{\text{init}}, P_r, y'_{<t}\big) \right]$

where $\pi_\theta(y'_t \mid x, y_{\text{init}}, P_r, y'_{<t})$ is the model's predicted probability of the $t$-th token of the revised response, conditioned on the problem $x$, the initial (possibly incorrect) response $y_{\text{init}}$, the control prompt $P_r$, and the previously generated tokens of the revision $y'_{<t}$.

What it computes: the standard autoregressive negative log-likelihood over the revised response tokens, but with the augmented context that includes the initial attempt and its outcome. This is a conditional generation loss: the model learns $p(\text{correct solution} \mid \text{problem}, \text{wrong attempt}, \text{"this is wrong"})$.

Why this form: maximizing the likelihood of the correct revision given the incorrect context teaches the model to (a) recognize what went wrong in the initial attempt by comparing it to the correct revision, (b) produce a corrected reasoning chain that avoids repeating the same errors, and (c) associate the control prompt "Wait, this response is not correct" with the action of starting over and finding the right answer. The model learns error correction as an explicit skill.

The generation loss retains the model's ability to produce correct responses from scratch, conditioned only on the problem. Critically, the target sequence here is not just $y'$ but the concatenation $[y_{\text{init}}, P_r, y']$:

$\mathcal{L}_{\text{generation}}(\theta) = \mathbb{E}_{(x, y_{\text{init}}, P_r, y') \sim \mathcal{D}_{\text{REVISION}}} \left[ -\sum_{t=1}^{|y'_{\text{concat}}|} \log \pi_\theta\big((y'_{\text{concat}})_t \mid x, (y'_{\text{concat}})_{<t}\big) \right]$

where $y'_{\text{concat}} = [y_{\text{init}}, P_r, y']$ — the concatenation of the initial attempt, the control prompt, and the correct revision, forming a single long sequence.

What it computes: the negative log-likelihood of the entire trajectory — initial attempt, acknowledgment of correctness/error via the control prompt, and final correct response — conditioned only on the problem. This is not a conditional generation loss; it's a standard autoregressive loss over a full self-revision trajectory.

Why this form: this is the mechanism that transfers revision capability into generation capability. By training the model to generate the full sequence $[y_{\text{init}}, P_r, y']$ when prompted only with $x$, the model learns to:

• Produce an initial attempt
• Evaluate that attempt (implicitly, by transitioning to "Wait, this is not correct" vs. "Let me rephrase")
• Generate a correction or rephrase

In other words, $\mathcal{L}_{\text{generation}}$ teaches the model to simulate the full revision process internally during a single forward pass, without needing an external verifier to trigger revision. This is what causes the SRT model to exhibit "explicit self-revision behaviors that lead to extremely long responses" (Section 2.1) — the model learns to generate both the flawed reasoning AND the correction in one go.

The combined SRT objective sums these two terms:

$\mathcal{L}_{\text{SRT}}(\theta) = \mathcal{L}_{\text{revision}}(\theta) + \mathcal{L}_{\text{generation}}(\theta)$

Why both terms are necessary: the ablation in Table 11 (Appendix E.2.1) demonstrates that neither term alone recovers full SRT performance. Using only $\mathcal{L}_{\text{generation}}$ preserves generation quality (65.4% first-attempt accuracy) but yields weak revision capability (correction rate only 7.2% vs. 15.0% for full SRT). Using only $\mathcal{L}_{\text{revision}}$ improves revision (correction rate 12.1%) but degrades generation substantially (62.1% first-attempt accuracy vs. 66.7% for full SRT). The two terms are complementary: $\mathcal{L}_{\text{revision}}$ directly teaches revision behavior by conditioning on the initial attempt, while $\mathcal{L}_{\text{generation}}$ teaches the model to internalize that revision behavior into autonomous single-pass generation. This complementarity echoes findings from Kumar et al. (2024b), which the paper cites: "self-correction does not emerge from a correction-focused signal alone, but requires preserving the model's underlying generation ability while separately training revision behavior."

Hyperparameters for SRT

The paper trains SRT on 6K self-revision traces (from 10K seed questions) with the hyperparameters reported in Table 4: AdamW optimizer, learning rate $5 \times 10^{-6}$, cosine scheduler with 0.05 warmup ratio, effective global batch size of 4, maximum sequence length of 32768 tokens, weight decay $1 \times 10^{-4}$, Adam betas (0.9, 0.95), bfloat16 precision, and 3 training epochs. Training uses "completion-only loss" — the loss is computed only on the response tokens, not on the input prompt tokens. Gradient checkpointing and FSDP (full_shard auto_wrap) are enabled for memory efficiency. Notably, the model is trained with thinking mode disabled ("Thinking: False" in Table 4) — the model generates non-thinking responses during both SRT and Self-Distillation, meaning the responses are concise reasoning chains rather than long exploratory chains of thought.

Phase 2 (Self-Distillation): On-Policy Self-Distillation via Revision Feedback

Phase 2 addresses a different problem: the SRT model, while accurate, produces overly long responses due to explicit self-revision behavior (average 8458 tokens for first attempts on AIME24, per Table 9). These long responses contain backtracking, "Wait, this is wrong" transitions, and full re-derivations — the model is effectively doing multi-turn revision in a single generation. The Self-Distillation phase distills the revision behavior back into the generator so the model can produce correct answers directly without simulating the revision process, reducing response length by roughly 2× while maintaining or improving accuracy.

The Student-Teacher Split

Phase 2 initializes from the SRT model and creates two roles from the same parameter set:

• Student (generator): parameters $\theta$, initialized as $\theta := \theta_{\text{SRT}}$, updated throughout Phase 2. The student generates on-policy responses $y \sim \pi_\theta(\cdot \mid x)$.
• Teacher (reviser): parameters $\theta_{\text{SRT}}$, frozen — no gradient updates. The teacher produces token-level targets by conditioning on the student's response and its binary reward: $\pi_{\theta_{\text{SRT}}}(\cdot \mid x, y, P_r)$.

The teacher's role is to provide a target distribution at every token position that represents what a "better" response would look like. For incorrect student responses, the teacher's distribution is conditioned on $P_r =$ "Wait, this response is not correct, let me start over" and the incorrect $y$, so the teacher's token-level probabilities reflect what it would generate as a correction. For correct student responses, the teacher conditions on the rephrase prompt, and its distribution reflects a rephrased (typically more concise) version.

The Self-Distillation Objective

For each training example, the student generates an on-policy response $y \sim \pi_\theta(\cdot \mid x)$. The binary verifier computes $r = r(y, a)$, and the control prompt $P_r$ is constructed. The student is trained to match the teacher's token-level distribution via KL divergence minimization:

$\mathcal{L}_{\text{Self-Distillation}}(\theta) = \mathbb{E}_{(x,a) \sim \mathcal{D}} \, \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)} \left[ \sum_{t=1}^{|y|} D_{\text{KL}}\Big( \pi_\theta(\cdot \mid x, y_{<t}) \;\big\|\; \pi_{\theta_{\text{SRT}}}(\cdot \mid x, y, P_r, y_{<t}) \Big) \right]$

where $\pi_\theta(\cdot \mid x, y_{<t})$ is the student's predicted next-token distribution given the problem and the partial response so far, and $\pi_{\theta_{\text{SRT}}}(\cdot \mid x, y, P_r, y_{<t})$ is the teacher's predicted next-token distribution given the problem, the complete student response $y$, the control prompt $P_r$, and the partial target tokens. The KL divergence $D_{\text{KL}}(P \| Q)$ measures how much the student's distribution $P$ diverges from the teacher's distribution $Q$ at each token position.

What it computes: for each token position $t$ in the student's response, the teacher — having seen the full student response and knowing whether it was correct or not — provides a probability distribution over the vocabulary that represents what the teacher thinks the "right" next token should be. The KL divergence penalizes the student for placing probability mass differently from the teacher. Summing over all token positions gives a total loss that pushes the student to generate responses whose token-level structure matches what the teacher would produce as a revision.

Why this form: KL divergence is the standard distillation loss because it preserves the full distributional information from the teacher, not just the single most likely token. This matters for two reasons. First, the teacher's distribution over alternative tokens may encode uncertainty or multiple valid continuations — forcing the student to match this uncertainty helps with generalization. Second, the KL divergence naturally handles the asymmetry between correct and incorrect student responses: for incorrect responses, the teacher's distribution will be sharply peaked around correction tokens (the "token-level self-localization" property), producing large gradients that push the student away from the erroneous tokens; for correct responses, the teacher's distribution is more uniform (the teacher is rephrasing, not correcting), producing gentler gradients that preserve the correct structure.

Why on-policy sampling matters: the expectation $\mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}$ means the student generates its OWN responses and receives feedback on them, rather than being trained on static pre-collected responses. This is critical because:

1. As the student improves during training, its responses change — the teacher must provide feedback on the current student's errors, not on errors the student made at initialization.
2. Off-policy training (using responses sampled from a different model) introduces distribution mismatch: the teacher's revision distribution is conditioned on the specific errors in those off-policy responses, which may not be representative of the student's actual error patterns.
3. The paper's related work section (Appendix A) cites prior evidence that off-policy distillation leads to "compounding errors at test time" (Bengio et al., 2015; Chen et al., 2025b), and that on-policy distillation addresses this by using student rollouts with per-token teacher supervision.

Detailed Per-Token Mechanism: Decomposing the KL Signal

The paper's Section 4.1 provides a detailed decomposition of the self-distillation loss that reveals how the sparse binary reward gets transformed into dense token-level supervision. Define the per-token KL divergence at position $t$:

$D^{(t)}_{\text{KL}} = D_{\text{KL}}\Big( \pi_\theta(\cdot \mid x, y_{<t}) \;\big\|\; \pi_{\theta_{\text{SRT}}}(\cdot \mid x, y, P_r, y_{<t}) \Big)$

This is the $t$-th term in the sum of the self-distillation loss. A related quantity, which the paper calls the Token KL Reward at token $t$, is defined as:

$\text{Token KL Reward}(t) = \log \pi_\theta(y_t \mid x, y_{<t}) - \log \pi_{\theta_{\text{SRT}}}(y_t \mid x, y, P_r, y_{<t})$

where $y_t$ is the actual token at position $t$ in the student's response, $\pi_\theta(y_t \mid \dots)$ is the student's probability assigned to that token, and $\pi_{\theta_{\text{SRT}}}(y_t \mid \dots)$ is the teacher's probability assigned to that token.

What it computes: the log-probability gap between student and teacher at a specific generated token. If the teacher assigns higher probability to the student's chosen token than the student itself does, the reward is positive — the teacher "agrees" with that token. If the teacher assigns lower probability, the reward is negative — the teacher would have preferred a different token. The per-token KL $D^{(t)}_{\text{KL}}$ aggregates this signal over the entire vocabulary.

Why this decomposition matters: Figure 4 (left) demonstrates that for incorrect student responses ($r = 0$), the per-token KL values are highly concentrated — a small number of tokens receive very large KL divergences, while most tokens receive negligible divergence. For correct student responses ($r = 1$), the KL values are much more uniform across tokens. This means the teacher is doing more than just assigning an overall "good/bad" score to the response: it is identifying which specific tokens are responsible for the error and focusing the gradient signal on those tokens.

Figure 4 (right) provides a concrete case study. For an incorrect geometry response that used a faulty symmetry-based argument, tokens associated with the symmetry claim receive large positive KL (teacher disagrees — these tokens are wrong), while tokens associated with a correct coordinate-based approach receive large negative KL (teacher agrees — these tokens should be retained). The teacher thus provides a two-sided token-level signal: it both penalizes erroneous reasoning tokens AND redirects probability mass toward correct alternative tokens.

This mechanism is what transforms the single-bit binary reward into dense token-level supervision. The reviser teacher, having been trained in Phase 1 to produce corrections when shown incorrect attempts, implicitly "knows" which parts of the student's response need to change. By conditioning on the full student response and the binary reward, the teacher produces a token-level distribution that encodes this knowledge. The student, trained to match this distribution, receives gradients that specifically target the tokens responsible for mistakes.

Hyperparameters for Self-Distillation

The paper trains Self-Distillation on 9K questions (the remaining questions from the 15K total, not used in Phase 1) with hyperparameters in Table 6: AdamW optimizer, learning rate $5 \times 10^{-6}$, linear warmup for 20 steps followed by constant schedule, effective global batch size of 128, temperature 1.0 for student sampling, top-K distillation with $K = 64$ (only the top-64 tokens in the teacher's distribution are used, with the rest zeroed out), maximum response length 8192 tokens, vLLM as the inference engine with tensor parallel size 4, and 4 GPUs. The teacher model is frozen — no gradients flow through it. The top-K distillation (K=64) is a practical optimization: computing the full vocabulary KL divergence would be expensive (vocabulary sizes ~50K-100K+), so only the 64 highest-probability tokens from the teacher are considered, and the student's distribution is renormalized over those tokens. This is a standard technique in distillation to reduce memory and computation.

Why Self-Distillation Improves Token Efficiency

The SRT model produces long responses (8458 average tokens on AIME24, Table 9) because it has learned to explicitly simulate the revision process: it generates an initial attempt, recognizes an error, backtracks with phrases like "Wait, this is wrong. Let me start over," and generates a corrected solution — all in one forward pass. The Self-Distillation phase teaches the model to internalize this behavior.

The mechanism is as follows. During SRT, the model learns that when it encounters a flaw in its own reasoning, the next tokens should be an acknowledgment + restart + correction. During Self-Distillation, the teacher (frozen SRT model) provides token-level targets that encode this pattern: for tokens where the student's response would have made an error, the teacher's distribution peaks at tokens that initiate correction. The student, trained to match this distribution via KL divergence, learns to anticipate the correction: rather than generating the error and then backtracking, it generates the correct reasoning directly, because the teacher's distribution at early token positions already reflects what the corrected reasoning would look like.

Figure 6 (Right) provides quantitative evidence for this transition. During SRT, both average response length and "self-revision keyword frequency" (ratio of tokens that are part of phrases like "Wait," "let me start over," etc.) rise. During Self-Distillation, both metrics fall — response length decreases by roughly 2×, and keyword frequency drops — while accuracy continues to improve. The paper interprets this as "a shift from overt revision to more internalized, token-efficient reasoning." The model no longer needs to explicitly backtrack; it has learned to produce reasoning that avoids the pitfalls that would have necessitated backtracking.

The qualitative example in Figure 6 (Left) illustrates this: the base model makes a false claim ("the smallest n such that every subset of size n contains three forming an isosceles triangle is 7"), the SRT model backtracks with "Wait, this is wrong. Let me start over" and finds the correct answer (6), while the SD-ZERO model, "without any backtracking, anticipates the same pitfall and directs proactively to the correct answer." The SD-ZERO model still achieves a 5.3% revision gain in the Generate-then-Revise evaluation (Figure 3, Table 9), showing that it has retained the revision capability — it just doesn't need to use it in single-pass generation anymore.

The Full Training Algorithm

Algorithm 1 in Appendix B.1 provides the pseudocode for the complete SD-ZERO pipeline. The procedure operates as follows:

Phase 1 (SRT):

1. For each training question, sample one initial response $y_{\text{init}}$ from the base model.
2. Compute the binary reward $r$ using the verifier.
3. Construct the control prompt $P_r$ based on correctness.
4. Generate a revised response $y_{\text{revised}}$ by conditioning on $(x, y_{\text{init}}, P_r)$.
5. Add the trace $(x, y_{\text{init}}, P_r, y_{\text{revised}})$ to $\mathcal{D}_{\text{REVISION}}$ only if $r(y_{\text{revised}}, a) = 1$.
6. Train the model on $\mathcal{D}_{\text{REVISION}}$ using $\mathcal{L}_{\text{SRT}}$ to obtain $\pi_{\theta_{\text{SRT}}}$.

Phase 2 (Self-Distillation): 7. Initialize student parameters $\theta \leftarrow \theta_{\text{SRT}}$. 8. Repeat for each training step:

• Sample a question $(x, a)$ from the Phase 2 dataset.
• Sample a student response $y \sim \pi_\theta(\cdot \mid x)$.
• Compute binary reward $r$ and construct control prompt $P_r$.
• Define student policy $\pi_S(\cdot) = \pi_\theta(\cdot \mid x)$ and teacher policy $\pi_T(\cdot) = \pi_{\theta_{\text{SRT}}}(\cdot \mid x, y, P_r)$.
• Update $\theta$ by minimizing $D_{\text{KL}}(\pi_S \| \text{StopGrad}(\pi_T))$.

9. Continue until convergence (1 epoch in the paper's experiments).

Optional: Teacher Synchronization: 10. After one epoch, set $\theta_{\text{SRT}} \leftarrow \theta$ (synchronize teacher with updated student). 11. Repeat steps 8–9 with the updated teacher.

The StopGrad operator in step 8 means gradients flow only through the student's parameters, not the teacher's — the teacher is frozen during each Self-Distillation epoch, providing stable targets.

Data Splits and Sampling Budgets

The paper uses 15K total training questions per domain (math: OpenR1-Math; code: Codeforces), split as follows (Appendix C.1):

• Phase 1 (SRT): 6K filtered self-revision traces, obtained from 10K seed questions. The 10K seed questions produce approximately 5K correct initial responses and 5K incorrect initial responses (the split depends on the base model's accuracy). Each initial response spawns 3 revision attempts = 30K total revision attempts, filtered to 6K successful revisions. Total generation budget for SRT data collection: 10K initial responses + 30K revisions = 40K generations.
• Phase 2 (Self-Distillation): 9K additional questions, each with 1 student rollout per training step. Total generation budget: 9K rollouts × 1 response = 9K generations.
• Total SD-ZERO generation budget: 40K + 9K = 49K generations.

The paper emphasizes in Appendix C.2 that this is comparable to or less than baseline methods under matched settings: RFT uses 15K × 4 = 60K generations, GRPO uses 15K × 4 = 60K generations (for the 4-rollout variant), and SDFT uses 15K × 4 = 60K generations. SD-ZERO achieves stronger performance with 49K total generations, and the Self-Distillation phase is especially sample-efficient because "it requires only one response per question, whereas SRT must sample multiple responses to construct revision traces" (Section 3.3).

The token budget analysis in Appendix C.2 estimates that SD-ZERO's total sampling budget is at most ~225M tokens (148M for SRT data collection + up to 76.5M for Self-Distillation), comparable to RFT/GRPO at ~222M tokens each. The training budget (tokens in the forward pass for loss computation) is at most ~219M tokens (66M for SRT + up to 153M for Self-Distillation).

Key Design Decisions and Their Justifications

Why two phases instead of one? The ablation in Table 12 (Appendix E.2.2) shows that applying Self-Distillation directly to the base model (without SRT) yields only marginal improvement: average generation accuracy rises from 49.8% to 51.4%, and the revision correction rate stays at 2.6% (vs. 2.7% for the base model). The reviser teacher has no revision capability to distill — it hasn't been trained to correct mistakes — so the token-level feedback is essentially noise. The SRT phase is necessary to first elicit self-revision ability; the Self-Distillation phase then refines and compresses it. This is a prerequisite relationship, not an optional enhancement.

Why filter revision traces for correctness? The paper provides direct evidence that unfiltered self-revision data degrades performance (Appendix G.1, Table 15). The mechanism is straightforward: if the model generates an incorrect initial response AND produces an incorrect revision, training on that trace teaches the model that "when you see this error pattern, produce this other error pattern" — teaching error-to-error transitions rather than error-to-correction transitions. The filtering step ensures the model only sees trajectories where revision succeeded, making the SRT objective a clean error-correction signal.

Why include both loss terms in SRT? The ablation in Table 11 (discussed above) demonstrates complementarity: $\mathcal{L}_{\text{revision}}$ alone teaches revision at the cost of generation quality; $\mathcal{L}_{\text{generation}}$ alone preserves generation but doesn't teach revision. The combined loss produces a model that can both generate from scratch AND revise when conditioned on a previous attempt. This dual capability is what enables Phase 2: the teacher must be able to produce good revisions when conditioned on student responses, and the student must be able to internalize that revision behavior into direct generation.

Why on-policy sampling in Phase 2? On-policy sampling ensures the teacher provides feedback on the current student's error patterns, not on errors from a previous model version. If the student improves during training, its error patterns change — off-policy data from an earlier checkpoint would not match the student's current distribution, and the teacher's revision targets (which are conditioned on specific errors in the student's response) would be misaligned. The paper cites prior work establishing that off-policy distillation leads to "compounding errors at test time" (Bengio et al., 2015; Chen et al., 2025b) as the theoretical motivation, and provides implicit evidence through the superior performance of the on-policy approach.

Why freeze the teacher during each Self-Distillation epoch? Freezing the teacher provides stable targets for the student. If the teacher were updated simultaneously with the student, the target distribution would drift during training, potentially leading to oscillations or collapse. The paper's iterative teacher synchronization approach (Section 3.4) provides a principled alternative: freeze the teacher for one full epoch, then synchronize with the improved student, enabling "iterative self-evolution" in discrete steps rather than continuous co-adaptation.

Why use two distinct control prompts ("rephrase" vs. "start over")? The two prompts serve different functions in the training pipeline. For correct responses, "rephrase" encourages the model to produce a different but still correct solution — this teaches the model that there are multiple valid reasoning paths and that conciseness is valued. For incorrect responses, "start over" explicitly acknowledges the error and instructs the model to produce a corrected solution from scratch rather than attempting to patch specific steps. This design choice avoids a subtle problem: if the prompt were simply "revise this response" for both correct and incorrect cases, the model might learn to always modify its response regardless of correctness, leading to the "correct-to-incorrect reversion problem" that plagued prior work on self-correction (Huang et al., 2023). The distinct prompts teach the model to condition on the reward: different actions for different outcomes. During inference in Phase 2, the teacher uses these distinct prompts based on the verifier's binary output, so the resulting token-level targets naturally encode "preserve if correct, redirect if incorrect."

Why is the SRT model's generation loss over the full concatenated trajectory? The concatenation $[y_{\text{init}}, P_r, y_{\text{revised}}]$ forces the model to learn the sequence of behaviors that constitutes self-revision: generate an attempt, evaluate it, generate a correction if needed. By training on this full sequence with only the problem as input, the model learns to produce the entire trajectory autonomously — which is exactly what produces the "explicit self-revision behavior" observed in the SRT model's long outputs. This is by design: the alternative of training only on $y_{\text{revised}}$ (ignoring the initial attempt and control prompt) would not teach the model to evaluate its own reasoning, only to produce correct answers. The concatenation approach makes the SRT model function as a "self-revising generator" that explicitly checks its work, while the subsequent Self-Distillation phase compresses this into implicit checking — the best of both worlds.

Connection to the Token-Level Self-Localization Property

Although Section 4.1 of the paper presents token-level self-localization as an analysis finding, it is better understood as a mechanism that explains why the Self-Distillation loss works. The teacher $\pi_{\theta_{\text{SRT}}}(\cdot \mid x, y, P_r)$ is not a generic "good response" distribution — it is a conditional distribution that knows the student's response $y$ AND whether it was correct. For incorrect $y$, the teacher has been trained (via SRT) to produce a correction, so its token-level distribution reflects two operations simultaneously:

1. Error localization: tokens in $y$ that correspond to the faulty reasoning receive higher KL divergence from the teacher's distribution — the teacher believes those tokens should have been different.
2. Correction redirection: at positions where the student is about to generate an erroneous token, the teacher's distribution concentrates mass on tokens that would lead toward the correct answer.

The visual evidence in Figure 4 supports this interpretation. The left panel shows that KL mass is concentrated on a small fraction of tokens for incorrect responses but spread uniformly for correct responses — the teacher is doing targeted intervention, not uniform enhancement. The right panel shows a specific example where tokens corresponding to a wrong symmetry argument receive positive KL (teacher disagrees) while tokens for a correct coordinate-based approach receive negative KL (teacher agrees). This is functionally equivalent to what a process reward model (PRM) would provide — per-step correctness signals — but without requiring step-level annotations or a separately trained reward model.

The paper draws this connection explicitly in Appendix A: "The per-token KL signal from the reviser is functionally analogous to process reward models (PRMs) (Lightman et al., 2024; Wang et al., 2024), in that both provide localized supervision over intermediate reasoning rather than only the final outcome." The key difference is cost: "PRMs typically require either step-level correctness annotations (Lightman et al., 2024) or rollout-based estimates from search or sampling (Wang et al., 2024), together with a separately trained reward model, whereas our reviser's signal arises by conditioning the same model on its own response and a binary outcome, without step-level annotation or an auxiliary reward model."

This connection matters because it positions SD-ZERO's mechanism within the broader landscape of test-time compute methods: if the reviser can provide PRM-like token-level feedback, then in principle that feedback could be used not just for distillation (training the generator) but also for guided search at inference time — an open direction the paper mentions.

4. Key Insights and Innovations

Innovation 1: The reviser as a mechanism for unpacking sparse rewards into dense token-level supervision — without a separately trained verifier

Prior to SD-ZERO, the field operated under an implicit dichotomy: you could have dense token-level supervision (via distillation from an external teacher, or from gold solution traces, or from a separately trained process reward model) OR you could have cheap binary reward signals (via RLVR methods like GRPO), but you could not have both simultaneously from a single model with only binary rewards. This was not a theoretical claim — it was simply the state of the art. On-policy distillation (Agarwal et al., 2024) required an external teacher. Self-distillation (OPSD, SDFT, SDPO) eliminated the external teacher but required privileged information — typically the correct solution — as context for the teacher. Process reward models (Lightman et al., 2024; Wang et al., 2024) provided dense step-level feedback but required either human annotations or Monte Carlo rollouts from a separately trained scoring model. GRPO and related RLVR methods required only binary rewards but provided no mechanism for converting those rewards into token-level feedback — the binary signal had to drive improvement through group-relative advantage comparisons across multiple rollouts.

SD-ZERO's central conceptual move is to recognize that a model trained to revise its own incorrect outputs can serve as the mechanism that unpacks binary rewards into dense token-level signal. The reviser is not a separate verifier model, nor does it receive privileged access to the correct answer. It receives exactly the same binary reward that GRPO receives — a single bit indicating correctness — plus the student's own response. What distinguishes the reviser from a generic teacher is that it has been trained (in Phase 1, SRT) specifically on the task of producing corrections when conditioned on incorrect attempts and their binary reward. That training imbues the reviser with a capability that the binary reward alone does not provide: the ability to identify which tokens in the student's response are problematic and what tokens should replace them.

This is a conceptual reframing rather than an incremental improvement. The dominant framing in RLVR is that the binary reward is the supervision signal, and the algorithm's job is to extract as much learning as possible from that sparse signal through clever credit assignment (advantage estimation, clipping, reference-model KL penalties). SD-ZERO reframes the binary reward not as the supervision signal itself, but as a conditioning variable that activates a learned revision capability which in turn produces the supervision. The binary reward becomes a switch that tells the reviser whether to correct or preserve, and the reviser's token-level distribution — conditioned on both the student's response and that switch — provides the actual training signal. This is a different category of solution: the model learns to generate its own dense supervision from sparse inputs, rather than learning to cope with sparse supervision.

The evidence for this reframing being meaningful rather than cosmetic comes from the token-level self-localization analysis in Section 4.1 (Figure 4). The reviser's per-token KL divergence is highly concentrated on a small fraction of tokens for incorrect responses, and nearly uniform for correct responses. This is not a property that emerges from any reasonable binary-reward RL objective — GRPO cannot produce concentrated token-level feedback because it has no mechanism for assigning credit to individual tokens. The reviser can because it was trained to generate corrections, and the act of generating a correction conditioned on an erroneous response implicitly requires deciding which tokens to keep and which to replace. The binary reward activates this pre-existing capability; it does not create it.

The significance of this reframing extends beyond the specific SD-ZERO algorithm. It suggests that any model trained on outcome-conditioned self-correction traces can, in principle, serve as a dense supervision generator for itself or for other models. The revision capability becomes a reusable resource: once a model learns to self-revise, it can provide token-level feedback to earlier versions of itself (as in iterative teacher synchronization, Section 3.4), to smaller models (as in standard distillation), or potentially to entirely different models trained on the same domain. The paper does not explore these extensions, but the conceptual framework enables them.

Innovation 2: Difficulty-appropriate supervision through outcome-conditioned teacher context

This innovation is subtler than the first and concerns what the teacher sees. In prior self-distillation work (OPSD, SDFT), the teacher conditions on privileged information that represents the "right answer" — typically a gold solution trace or a high-quality demonstration. This means the teacher knows the correct answer before providing feedback, and its token-level targets are shaped by that knowledge. The feedback is essentially: "here's how the correct solution would look at this token position, try to match it."

SD-ZERO's teacher conditions on something fundamentally different: the student's own response and its outcome (correct/incorrect). When the student's response is wrong, the teacher does not have access to the correct answer — it must infer the correction from the student's flawed reasoning and the knowledge that the reasoning is flawed. This is a substantially harder task for the teacher, but it produces feedback that is more closely aligned with the student's actual error patterns. The teacher isn't saying "here's the right answer, match it"; it's saying "given that THIS specific reasoning chain is wrong, here's what a corrected version would look like."

This matters for a reason that connects back to the difficulty-dependent findings from the prior paper (Snell et al., "Scaling LLM Test-Time Compute Optimally"): the quality of supervision depends on how well-matched it is to the learner's current capabilities. A gold solution trace provides perfect supervision in an information-theoretic sense, but it may be so far from the student's current distribution that the student cannot effectively learn from it — the gradient signal is large but uninformative because the student's errors are so different from anything the gold solution addresses. By contrast, the reviser's correction is generated from the student's actual attempt: it keeps as much of the student's reasoning as possible and only changes the parts that need changing. This makes the gradient signal both smaller in magnitude and more targeted — the student is asked to change only the specific reasoning steps that led to the error, not to reproduce an entirely foreign solution.

The evidence for this being a meaningful distinction comes from the SDFT-with-final-answers-only ablation in Table 8. When SDFT's teacher is given only the ground-truth final answer (not the full solution trace) — which is arguably a closer analog to what SD-ZERO's teacher receives — SDFT performs "only marginally better than the base model" and "remains far below SD-ZERO." This suggests that SDFT's gains come primarily from having access to the full solution trace as privileged context, not from the self-distillation mechanism itself. When that privileged context is reduced to the same information SD-ZERO receives (a binary outcome), the self-distillation becomes ineffective. SD-ZERO's reviser, by contrast, is specifically designed to operate with exactly that level of information — it learned in Phase 1 to generate corrections from binary outcomes, so in Phase 2 it can provide useful token-level feedback without needing privileged access to solutions.

This is not merely a "weaker supervision works better" finding — that would be surprising and likely domain-specific. Rather, it's that the form of supervision matters as much as its quality. A reviser that generates corrections from the student's actual errors provides structurally different feedback from a teacher that reproduces a pre-existing solution. The former teaches error correction; the latter teaches solution reproduction. On tasks where error patterns are systematic (certain types of reasoning mistakes recur across problems), learning to correct errors may generalize better than learning to reproduce solutions that didn't address those specific errors. This connects to the observation in Appendix G.1 that on-policy data — even when generated by the same model that will be trained on it — outperforms off-policy data from stronger teachers. The mechanism proposed here (error-specific feedback rather than generic correct-solution feedback) provides a possible explanation.

Innovation 3: Self-distillation as a compression mechanism that internalizes multi-step revision into single-pass generation

The paper documents a specific behavioral transformation during the Self-Distillation phase: the SRT model generates long responses (8458 average tokens on AIME24) with explicit self-revision markers ("Wait, this is wrong. Let me start over"), while the SD-ZERO model generates responses roughly half as long (~3500 tokens, Table 9) with no explicit backtracking, yet achieves higher accuracy. This is not merely an efficiency improvement — it represents a different mode of reasoning.

The conceptual innovation is recognizing that distillation from a reviser teacher does not just improve accuracy; it changes how the model reasons. The SRT model reasons by generating, evaluating, backtracking, and regenerating — an explicit multi-step process compressed into a single autoregressive generation. The SD-ZERO model reasons by anticipating errors and avoiding them proactively — an implicit process that produces correct reasoning on the first pass without needing to backtrack.

This distinction matters because it connects to a broader question about language model reasoning: do models improve by learning to detect and fix errors (an error-correction capability), or by learning to avoid errors in the first place (a reasoning-quality improvement)? The SRT phase teaches error correction — the model learns to fix mistakes when it makes them. The Self-Distillation phase transforms error correction into error avoidance — the model internalizes the correction patterns so thoroughly that it no longer makes the mistakes that would require correction.

Figure 6 provides the quantitative evidence for this transformation: response length and self-revision keyword frequency both rise during SRT (the model learns to explicitly revise) and fall during Self-Distillation (the model internalizes revision) while accuracy continues to improve. The qualitative example confirms this: the SD-ZERO model "anticipates the same pitfall and directs proactively to the correct answer" rather than falling into the pitfall and climbing out.

This is significant beyond the specific SD-ZERO pipeline because it suggests a general principle: explicit reasoning behaviors (verification, backtracking, correction) can be trained into models via supervised fine-tuning on demonstration traces, and then compressed into implicit reasoning capabilities via distillation. This is analogous to how human experts internalize checking procedures that novices must perform explicitly — the expert "just knows" the answer is right without consciously verifying, not because they skip verification, but because their primary reasoning process has become reliable enough that explicit checking is redundant. The Self-Distillation phase achieves this compression through KL-divergence minimization: by forcing the student to match the teacher's token-level distribution without actually generating the backtracking tokens, the student learns to jump directly to the corrected reasoning.

This insight also explains why the Self-Distillation phase is "critical for the training and inference efficiency of SD-ZERO" (Section 1, Key Findings): without it, the SRT model would be accurate but impractical for deployment due to the 2× token overhead. With it, the model achieves both better accuracy AND better efficiency — the distilled model is strictly superior on both axes. This is rare in machine learning, where accuracy-efficiency tradeoffs are the norm, and it suggests that the Self-Distillation phase is doing something more fundamental than simply "training on more data" — it is changing the model's reasoning strategy.

Innovation 4: Iterative self-evolution through teacher synchronization — a capability amplifier, not a one-shot recipe

The final conceptual innovation concerns the dynamic of SD-ZERO rather than its static performance. The paper demonstrates in Section 3.4 (Figure 5) that after one epoch of Self-Distillation, the improved student can replace the frozen teacher, and a second round of distillation yields an additional 3+ percentage point gain "without signs of saturation." This is not merely a "train longer" result — it represents a specific feedback loop that the paper terms "iterative self-evolution."

The loop works as follows: Phase 1 (SRT) establishes a baseline revision capability. Phase 2 (Self-Distillation) uses that revision capability to improve generation. The improved generation implies improved revision (because the model that generates better answers can also produce better corrections when shown incorrect attempts). The improved revision capability can then be used as a stronger teacher for a new round of Self-Distillation. Each turn of this crank improves both generation and revision, with the improvement in each feeding the other.

What makes this distinctive from standard iterative training approaches (like STaR's bootstrapping loop, or the repeated generate-filter-train cycles in self-training) is the role of the revision capability as the transmission mechanism. In standard self-training, the model generates responses, filters for correctness, and trains on the correct ones — the model improves because it gets more training data, but the quality of each round's data depends only on the model's generation capability. In SD-ZERO, the revision capability acts as an amplifier: even when the generator produces incorrect responses, the reviser can convert those incorrect responses into training signal (via the Self-Distillation KL objective), so the effective amount of supervision per generated response is higher than in standard self-training. As the generation capability improves, the revision capability improves with it (because the same model does both), creating a virtuous cycle.

The evidence that this is a genuine capability-amplification effect rather than simply "more training" comes from the Generate-then-Revise evaluation in Figure 3: the SD-ZERO model achieves a 5.3% revision gain (comparable to SRT's 5.0%), meaning its revision capability has been preserved or improved despite the Self-Distillation phase compressing its explicit revision behavior. The model has gotten better at both generating correct answers directly AND revising incorrect ones when needed — the two capabilities co-evolve rather than trading off.

This positions SD-ZERO not as a fixed training recipe but as a priming mechanism: once a model has been initialized with revision capability (via SRT), it can enter a self-reinforcing improvement loop that requires only binary rewards and on-policy sampling, with no external teacher, no gold solutions, and no step-level annotations. The paper demonstrates only two rounds of this loop, but the lack of saturation in the second round suggests that further rounds would continue to yield gains, making SD-ZERO a candidate for open-ended self-improvement in verifiable domains.

The connection to the "compute-optimal test-time scaling" paper is worth noting: that paper found that test-time compute can substitute for pretraining compute up to a difficulty ceiling determined by the base model's capabilities. SD-ZERO's iterative self-evolution can be seen as a mechanism for raising that ceiling — each round of teacher synchronization effectively improves the base model's revision capability, which improves its generation capability, which may expand the set of problems for which test-time compute (in the form of revision-based correction at inference) is effective. The paper does not explore this connection, but the conceptual framework supports it.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper trains on two domains separately: (1) OpenR1-Math (Hugging Face, 2025), selecting 15K competition- and olympiad-level problems with verified solutions, and (2) Codeforces (Penedo et al., 2025), selecting 7.5K from the cpp (C++ solutions) subset and 7.5K from the Python (solutions_py) subset totaling 15K code problems. Training data curation details are in Appendix C.1. For evaluation, the paper holds out 500 test questions from each of the two in-distribution datasets and additionally evaluates on six out-of-distribution benchmarks to test generalization.

• Base model(s). All experiments use two model families: Qwen3-4B-Instruct (Yang et al., 2025), a 4-billion-parameter instruction-tuned model, and Olmo-3-7B-Instruct (Olmo et al., 2025), a 7-billion-parameter instruction-tuned model. The authors "use Qwen3-4B-Instruct and Olmo-3-7B-Instruct as base models" (Section 3.1) because they represent capable and widely available open-weight instruction-tuned models at moderate scale, making the experiments reproducible and the results relevant to practitioners with limited compute. Both models are used with thinking mode disabled ("Thinking: False" in Table 4), meaning they produce concise reasoning chains rather than long exploratory chain-of-thought responses.

• Metrics. The primary metric throughout is accuracy, specifically avg@8 — the average accuracy across 8 independent samples per question, where a response is counted as correct if its extracted final answer matches the ground truth. For the in-distribution evaluation sets (OpenR1-Math and Codeforces), the paper reports accuracy on the held-out 500 test questions. For out-of-distribution benchmarks, standard accuracy metrics are used. Pass@8 is also reported in Table 7 as a complementary metric to assess whether the model's improvements reflect genuine capability expansion rather than distribution sharpening. For the Generate-then-Revise analysis (Section 3.2, Figure 3), the paper additionally measures First Attempt Accuracy, Revised Attempt Accuracy, and Correction Rate — defined as (Revised Attempt Accuracy − First Attempt Accuracy) / (100 − First Attempt Accuracy), which is the fraction of initially incorrect answers that are successfully corrected after revision.

• Baselines. The paper compares against four post-training methods, all trained on the same 15K questions under matched generation budgets (as documented in Appendix C.2, Table 3):

  • SFT (Supervised Fine-Tuning): fine-tuning on high-quality demonstrations from DeepSeek-R1 (Guo et al., 2025). This represents training on external teacher data at the token level.
  • RFT (Rejection Fine-Tuning): fine-tuning on only the correct self-generated responses from the base model (Yuan et al., 2023). Incorrect responses are discarded entirely.
  • GRPO (Group Relative Policy Optimization): reinforcement learning with binary correctness reward, using the DAPO variant (Yu et al., 2025) which is "an improved and commonly used variant of GRPO" (Section 3.1). The paper's main GRPO configuration uses 4 rollouts per question and 1 training epoch.
  • SDFT (Self-Distillation Fine-Tuning): on-policy self-distillation where the model conditions on high-quality demonstrations (gold solution traces) as its own teacher (Zhao et al., 2026b; Shenfeld et al., 2026b), using 4 rollouts per question and 1 training epoch.

  SD-ZERO is compared in two forms: SRT (Phase 1 only, Self-Revision Training on 6K traces) and SD-ZERO (full two-phase pipeline, SRT + Self-Distillation). All methods are evaluated under a matched total generation budget, with detailed accounting provided in Appendix C.2 (Table 3).

• Generation budget / compute accounting. The paper normalizes all comparisons by the total number of model-generated responses across all phases of training. As detailed in Appendix C.2 (Table 3): RFT, GRPO, and SDFT each use 15K questions × 4 rollouts = 60K total generations. SD-ZERO uses 40K generations for SRT data collection (10K initial responses + 30K revisions) + 9K generations for Self-Distillation rollouts = 49K total generations. The paper also estimates total token budgets: RFT/GRPO at ~222M tokens, SD-ZERO at ≤225M tokens (Appendix C.2), confirming that the generation budgets are comparable and that SD-ZERO's advantages are not due to using more compute. The Self-Distillation phase's efficiency is specifically highlighted: it "requires only one response per question, whereas SRT must sample multiple responses to construct revision traces" (Section 3.3). All sampling during training uses temperature 0.7 with a 16K token limit; evaluation uses temperature 0.7 with a 32K token limit (Section 3.1).

• Cross-validation / statistical protocol. The paper does not use formal cross-validation on the 15K training set; instead, it performs a simple hold-out by reserving 500 test questions from each of the in-distribution datasets (OpenR1-Math and Codeforces) and reports results on those held-out sets as well as on six out-of-distribution benchmarks. The data split between Phase 1 (6K traces from 10K seed questions) and Phase 2 (9K additional questions) is fixed in the main experiments, with an ablation on the split ratio provided in Table 13. For the GRPO baseline, the paper additionally sweeps different rollout budgets and training epochs (4 generations/1 epoch, 8 generations/0.5 epoch, 8 generations/1 epoch) in Table 10, finding that "simply increasing the rollout budget in GRPO does not close the gap to our method." All experiments are conducted once per configuration without reporting confidence intervals or standard deviations, which is a limitation of the experimental design.

Main Quantitative Results

Phase 1 (SRT) Outperforms All Baselines With 2.5× Fewer Training Examples

The headline result for Phase 1 appears in Table 1: SRT, trained on only 6K self-revision traces (compared to 15K examples for all baselines), achieves the best or second-best performance across nearly all benchmarks for both model families.

For Qwen3-4B-Instruct, SRT achieves an average accuracy of 57.6% across all 8 benchmarks, compared to:

• Base model: 49.8% (+7.8 percentage points)
• SFT (on DeepSeek-R1 traces): 50.0% (+0.2 over base — essentially no gain)
• RFT (on filtered correct self-generations): 54.3% (+4.5 over base)
• GRPO (4 rollouts, 1 epoch): 53.1% (+3.3 over base)
• SDFT (on-policy self-distillation with gold solutions): 51.2% (+1.4 over base)

The most striking SRT gains appear on the hardest benchmarks. On AIME25, SRT reaches 59.2% vs. 52.1% for RFT (the next best baseline), a +7.1 point gap. On HMMT25, SRT achieves 40.0% vs. 37.1% for RFT. On LiveCodeBench, SRT reaches 74.4% vs. 68.0% for RFT, a +6.4 point gap. On AMOBench, SRT achieves 16.0%, while RFT reaches only 11.3% and SFT actually degrades the base model (9.8% → 7.3%). This pattern — where SRT's advantage is largest on the hardest benchmarks — is consistent with the paper's argument that learning from mistake-to-correction trajectories is most valuable when the problems are difficult enough that the base model frequently makes mistakes.

For Olmo-3-7B-Instruct, the pattern is similar: SRT achieves an average of 50.3% across all benchmarks, compared to 41.1% for the base model (+9.2 points) and 46.7% for RFT (+3.6 point gap). On AIME25, SRT reaches 52.9% vs. 48.3% for RFT. On HMMT25, 39.6% vs. 35.8%. On LiveCodeBench, 59.6% vs. 49.4%.

The paper attributes SRT's effectiveness to its use of both successful and unsuccessful reasoning in the training data: "Unlike RFT, which removes incorrect reasoning entirely, SRT preserves the failed attempt as context, allowing the model to learn from its own mistakes" (Section 3.2). Table 1 provides the quantitative evidence: across all benchmarks where the base model has non-trivial accuracy (so there are mistakes to learn from), SRT substantially outperforms RFT, which trains on the same underlying model generations but discards the incorrect ones.

Phase 2 (Self-Distillation) Adds Further Gains While Compressing Responses 2×

The full SD-ZERO pipeline (SRT + Self-Distillation) achieves the strongest results across all comparisons in Table 1.

For Qwen3-4B-Instruct, SD-ZERO reaches an average of 60.3% across all 8 benchmarks, which is:

• +10.5 points over the base model (49.8%)
• +2.7 points over SRT alone (57.6%)
• +6.0 points over RFT (54.3%), the strongest baseline
• +7.2 points over GRPO (53.1%)

The Self-Distillation phase provides uneven gains across benchmarks. The largest improvements come on code tasks: Codeforces jumps from 74.4% (SRT) to 82.6% (SD-ZERO), a +8.2 point gain, and LiveCodeBench rises from 52.7% to 56.1% (+3.4 points). On math, HMMT25 benefits substantially (40.0% → 45.4%, +5.4 points), while AIME24 and AIME25 show more modest gains (66.7% → 68.3% and 59.2% → 60.0% respectively). This heterogeneity suggests the Self-Distillation phase is most valuable when the SRT model's explicit self-revision behavior produces particularly verbose outputs that benefit from compression — code generation, with its multi-step implementation and debugging, may be a domain where distillation compresses substantially more token overhead.

For Olmo-3-7B-Instruct, SD-ZERO reaches an average of 51.5%, which is:

• +10.4 points over the base model (41.1%)
• +1.2 points over SRT alone (50.3%)
• +4.8 points over RFT (46.7%)

The Self-Distillation phase helps Olmo primarily on math benchmarks: AIME24 (59.2% → 61.7%), HMMT25 (39.6% → 40.4%), and AMOBench (3.5% → 5.5%). Code gains are more modest or even slightly negative (Codeforces: 59.6% → 57.8%), suggesting that the benefit of Self-Distillation depends on the base model's specific characteristics and error patterns.

Pass@8 results in Table 7 confirm that SD-ZERO's gains are not merely sharpening the output distribution around a narrow set of solutions. For Qwen3-4B-Instruct, Pass@8 average across math benchmarks reaches 72.5% for SD-ZERO vs. 70.4% for SRT and 69.8% for RFT. The pass@8 improvement suggests that "our method is not merely sharpening the output distribution around a narrow set of solutions, but genuinely making the model's exploration more well-directed, so that correct reasoning paths are more likely to be discovered across multiple attempts" (Appendix D.1). This contrasts with GRPO, which Yue et al. (2025) suggest "mainly sharpen the output distribution" — consistent with GRPO's Pass@8 of 65.4% on math benchmarks for Qwen3-4B, which is substantially below SD-ZERO's 72.5%.

Self-Revision Capability: SRT Dramatically Improves Error Correction

The Generate-then-Revise evaluation on 1K AIME24 questions with Qwen3-4B-Instruct (Figure 3, detailed in Table 9) directly measures the model's ability to correct its own mistakes when explicitly prompted to revise.

The base model gains only +1.1 percentage points from revision (59.6% → 60.7%), with a correction rate of 2.7% — meaning only 2.7% of initially incorrect answers are fixed by revision. This confirms the paper's premise that "our base models exhibit strong generator performance but weak reviser performance" (Section 2.1).

SRT dramatically changes this. First-attempt accuracy rises to 66.7%, and revised-attempt accuracy reaches 71.7% — a +5.0 point gain from revision, with a correction rate of 15.0%. This is a 5.6× improvement in correction capability over the base model. The paper notes that "the SRT model has learned to use outcome reward to correct its first attempt rather than simply resampling a new attempt" (Section 3.2).

Surprisingly, the revision responses from SRT are shorter than first attempts (8137 vs. 8458 tokens, Table 9). This is the opposite pattern from the base model, where revisions are longer than first attempts (5098 vs. 3708). The paper interprets this as evidence that SRT's revisions are "more targeted and proactive": the model focuses on fixing the specific error rather than regenerating the entire solution verbosely.

SD-ZERO further improves on these metrics: first-attempt accuracy reaches 68.3%, revised-attempt accuracy 73.6%, and correction rate 16.7%. Notably, both first-attempt and revision responses from SD-ZERO are much shorter (~3500 tokens vs. SRT's ~8300) — the Self-Distillation phase has compressed the reasoning without losing the revision capability, as evidenced by the maintained correction rate (16.7% vs. SRT's 15.0%).

Comparisons to baselines on this metric are informative: GRPO achieves only a 4.9% correction rate and SDFT only 3.8% (Table 9), confirming that neither RL-based training nor standard self-distillation teaches the model to effectively correct its own errors. The baseline methods' revision attempts tend to be longer than first attempts (GRPO: 4432 → 5499; SDFT: 3630 → 5099), suggesting they generate redundant content rather than targeted corrections.

Response Length Compression: Self-Distillation Reduces Token Cost by ~2×

The response length dynamics are tracked in Figure 6 (Right) and quantified in Table 9. For Qwen3-4B-Instruct on AIME24:

• Base model: 3708 average tokens (first attempt)
• SRT model: 8458 average tokens — 2.3× longer than base
• SD-ZERO model: 3518 average tokens — actually shorter than the base model, and 2.4× shorter than SRT

The SRT model's verbosity is by design: during SRT, "both metrics rise substantially. The model learns to solve problems by actively judging its own response and applying self-correction" (Section 4.2). The SD-ZERO model's conciseness is achieved through distillation: "during Self-Distillation phase, the trend reverses: both response length and self-revision keyword frequency fall steadily, while accuracy continues to improve."

Figure 3 places this in context against baselines, showing that SD-ZERO achieves the best accuracy among all methods while having the lowest token cost per question (~3500 tokens vs. ~3700–4400 for baselines and ~8500 for SRT). This is a rare case where the strongest model is also the most efficient — there is no accuracy-efficiency tradeoff.

The mechanism for this compression is visible in Figure 6 (Left), which provides qualitative examples: the base model makes a false claim without revision; the SRT model backtracks with "Wait, this is wrong. Let me start over" and regenerates; the SD-ZERO model "anticipates the same pitfall and directs proactively to the correct answer" without any backtracking. The paper characterizes this as "a shift from overt revision to more internalized, token-efficient reasoning" (Section 4.2).

Comparison to GRPO Under Matched and Expanded Budgets

The main comparison in Table 1 uses GRPO with 4 rollouts per question and 1 training epoch, matched to SD-ZERO's total generation budget. SD-ZERO outperforms GRPO by 7.2 points on average for Qwen3-4B-Instruct (60.3% vs. 53.1%) and by 6.7 points for Olmo-3-7B-Instruct (51.5% vs. 44.8%). The Gap is particularly large on the hardest benchmarks: AIME25 (60.0% vs. 50.0% for Qwen3-4B; 53.8% vs. 43.8% for Olmo-3-7B), HMMT25 (45.4% vs. 30.4% for Qwen3-4B; 40.4% vs. 25.8% for Olmo-3-7B), and LiveCodeBench (82.6% vs. 62.6% for Qwen3-4B; 57.8% vs. 43.6% for Olmo-3-7B).

Table 10 explores whether giving GRPO more generation budget closes this gap. For Qwen3-4B-Instruct on math benchmarks:

• GRPO (4 generations, 1 epoch): 51.7% average
• GRPO (8 generations, 0.5 epoch): 50.6% — slightly worse, likely due to fewer gradient steps
• GRPO (8 generations, 1 epoch): 52.3% — a modest +0.6 point improvement with 2× the sampling budget as SD-ZERO
• SD-ZERO: 57.3% average on math

The paper notes that "even when GRPO is given substantially more exploration through 8 generations, its gains remain limited, whereas SD-Zero achieves the best average performance and outperforms all GRPO settings on most benchmarks" (Table 10 caption). Moving from 4 to 8 generations with a full epoch provides only a +0.6 point gain on average (51.7% → 52.3%), while SD-ZERO outperforms by +5.0 points on the same math benchmark subset. This strongly suggests that GRPO's limitation is not sample quantity but supervision quality — the binary reward provides insufficient per-sample information, and increasing the number of samples cannot compensate for this fundamental sparsity.

The paper emphasizes that the Self-Distillation phase's efficiency advantage over GRPO is structural: "unlike GRPO, whose training requires a group of sampled responses per question, the Self-Distillation phase of SD-ZERO uses only a single response per question, substantially reducing generation cost" (Section 3.3). This is a direct consequence of the reviser converting binary rewards into dense token-level feedback: each sample provides gradients at every token position, compared to GRPO's group-relative advantage signal which requires multiple samples per question to estimate which responses are better than others.

SDFT With Final-Answer-Only Supervision Fails to Match SD-ZERO

Table 8 demonstrates that SD-ZERO's gains are not achievable by simply giving SDFT the same level of supervision. When SDFT's teacher is provided only the ground-truth final answer (not the full gold solution trace), performance on Qwen3-4B-Instruct across math benchmarks averages 49.5% — nearly identical to the base model's 49.7% and far below SD-ZERO's 57.3%. The full SDFT (with gold solutions) achieves only 49.7%, representing no improvement over the final-answer-only variant.

This ablation is critical because it isolates what makes SD-ZERO's mechanism different from SDFT. Both methods use on-policy self-distillation with a self-teacher, but SDFT's teacher conditions on privileged information (a correct solution) while SD-ZERO's reviser conditions on the student's response and its binary outcome. When SDFT is denied the privileged information and given only the binary outcome, its self-distillation becomes ineffective. SD-ZERO, by contrast, is specifically designed to operate in this setting — the SRT phase teaches the reviser to generate corrections from binary outcomes, so in Phase 2 the teacher can produce useful token-level feedback from just the student's response and its correctness. This confirms that SD-ZERO's contribution is not "self-distillation works" but specifically "self-distillation can work when the teacher is trained to convert binary rewards into dense token-level signal."

Iterative Self-Evolution: Teacher Synchronization Yields Continued Gains

Figure 5 demonstrates that SD-ZERO's gains are not bounded by the quality of the initial SRT teacher. On OpenR1-Math with Qwen3-4B-Instruct, the first Self-Distillation phase saturates around training step 400 at approximately 57% accuracy. After synchronizing the teacher with the updated student (setting $\theta_{\text{SRT}} \leftarrow \theta$) and continuing training, a second phase yields "at least 3 additional percentage points without signs of saturation" (Section 3.4).

This result establishes that SD-ZERO is not merely a one-shot improvement over the base model but a capability amplifier: the improved student becomes a stronger teacher, which further improves the student in the next round. The paper frames this as "iterative self-evolution" where "the capability to revise answers is distilled back into generation, and regular teacher synchronization allows the loop to sustain" (Section 3.4).

The mechanism enabling this loop is that Self-Distillation improves not just generation but also revision — as evidenced by the 5.3% revision gain for SD-ZERO vs. 5.0% for SRT in the Generate-then-Revise evaluation (Figure 3, Table 9). The improved revision capability means the synchronized teacher can provide even more accurate token-level feedback in the next round.

The paper notes that this loop "requires only the initial attempt and corresponding binary reward in the teacher's context" (Section 3.4), meaning it can continue indefinitely without additional human annotation or external resources — each round uses only the model's own on-policy samples and binary verifier outputs.

Ablation Studies and Robustness Checks

Loss terms in SRT objective (Table 11): The SRT loss combines $\mathcal{L}_{\text{generation}}$ and $\mathcal{L}_{\text{revision}}$. Training with only $\mathcal{L}_{\text{generation}}$ preserves strong first-attempt accuracy (65.4% on AIME24) but yields weak revision (correction rate 7.2% vs. 15.0% for full SRT). Training with only $\mathcal{L}_{\text{revision}}$ achieves decent revision (correction rate 12.1%) but substantially degrades first-attempt generation (62.1% vs. 66.7% for full SRT). Neither single-term variant matches the full SRT's overall generation accuracy (56.4% and 52.2% respectively vs. 57.6%). The paper concludes these terms are "complementary: $\mathcal{L}_{\text{revision}}$ elicits self-revision, and $\mathcal{L}_{\text{generation}}$ transfers it to stronger generation" (Appendix E.2.1). This finding aligns with prior work (Kumar et al., 2024b) showing that correction ability requires preserving generation ability while separately training revision.

Self-Distillation without SRT phase (Table 12): Applying the Self-Distillation phase directly to the base Qwen3-4B-Instruct model (without Phase 1) yields only marginal improvement: average generation accuracy rises from 49.8% to 51.4%, and the revision correction rate remains at 2.6% (essentially unchanged from the base model's 2.7%). Generate-then-Revise first-attempt accuracy inches from 59.6% to 61.2%, and revised-attempt accuracy from 60.7% to 62.2%. The paper concludes that "the SRT phase appears to be a necessary prerequisite that first elicits self-revision behavior, after which the Self-Distillation phase can effectively refine and strengthen it" (Appendix E.2.2). This confirms that the reviser teacher must actually possess revision capability for the distillation to provide useful token-level feedback — a frozen base model asked to generate revisions provides essentially random targets.

Data split between SRT and Self-Distillation phases (Table 13): When keeping the total training data fixed and varying the allocation between Phase 1 (SRT) and Phase 2 (Self-Distillation), three splits are tested on Qwen3-4B-Instruct: (6K SRT, 9K Self-Distillation), (9K SRT, 6K Self-Distillation), and (7.5K, 7.5K). Allocating more data to SRT modestly improves the SRT model (57.8% for 9K/7.5K vs. 57.6% for 6K) but slightly degrades the final SD-ZERO model (59.1% for 9K/6K vs. 60.3% for 6K/9K). The best final performance comes from giving more data to Phase 2, suggesting that "once SRT has unlocked self-revision capability, Self-Distillation phase uses additional data more effectively to distill and refine this behavior into stronger generation" (Appendix E.2.3). The practical implication is that SRT needs only enough data to establish baseline revision capability (6K traces appears sufficient); additional data is better spent on distillation.

SRT with unfiltered revision traces (Appendix G.1, Table 15): On the Countdown task with Qwen2.5-7B, the paper compares filtered self-revision (pass@1 = 0.630) against unfiltered self-revision (pass@1 = 0.529), finding that "noisy revision traces actively hurt" — a 19% relative degradation from omitting correctness filtering. This validates the design choice in Phase 1 to retain only revision traces where the revised response is correct. The negative result is informative: it demonstrates that training on incorrect revisions teaches the model error-to-error transitions rather than error-to-correction transitions.

GRPO configuration comparison (Table 10): As discussed in the main results above, increasing GRPO's rollout budget from 4 to 8 generations per question (with matched or doubled training epochs) provides at most a +0.6 percentage point improvement on average (51.7% → 52.3%), while still falling far below SD-ZERO's 57.3%. This ablation confirms that GRPO's sample inefficiency is not simply a matter of insufficient samples per update — even with 2× the sampling budget, the sparse binary reward signal cannot match the dense token-level feedback from the reviser.

SDFT with thinking models (Table 14): Appendix F explores extending self-distillation to "thinking models" that generate long, exploratory chains of thought. When SDFT is applied to Qwen3-4B with thinking enabled during the student's on-policy rollouts, performance degrades substantially on competition math benchmarks: −9.8 points on AIME24, −10.8 on AIME25, and −8.3 on HMMT25 compared to the base model. Disabling thinking during training largely preserves (and on HMMT25 slightly improves) the base model's accuracy. This negative result is significant because it demonstrates that "extending self-distillation to thinking models... is challenging because such responses may include false starts and partial corrections that are not mistakes, making it difficult to distinguish productive exploration from genuine errors and assign credit beyond local token decisions" (Section 5). This motivates the paper's focus on "instruct models that generate short and concise responses" and highlights that the effectiveness of the self-distillation mechanism depends on the student generating responses whose tokens can be cleanly categorized as correct or incorrect.

Data source comparison for self-revision training (Appendix G.1, Table 15): On the Countdown task with Qwen2.5-7B, the paper compares SFT on various data sources: off-policy data from LLaMA-70B (pass@1 = 0.605), off-policy data from GPT-4o (0.617), on-policy filtered first attempts (0.552), on-policy filtered self-revision (0.630), and SFT on GRPO-trained model outputs (0.878–0.883). Two findings relevant to SD-ZERO: First, filtered self-revision (0.630) substantially outperforms filtered first attempts (0.552), confirming that the revision step itself adds value beyond simply using the model's correct first attempts. Second, off-policy data from stronger teachers (GPT-4o at 0.617) underperforms on-policy self-revision (0.630), suggesting that "distribution match matters more than teacher quality" for SFT on reasoning tasks. SD-ZERO's design leverages both insights: Phase 1 uses on-policy self-revision (not off-policy teacher data), and Phase 2 uses the model as its own teacher (not an external one).

Self-revision as initialization for RL (Appendix G.2, Table 16): On out-of-distribution benchmarks (AIME24, AIME25), applying GRPO after SFT on self-revised targets (A′+GRPO) achieves pass@128 of 0.40 on AIME24 and 0.47 on AIME25, compared to 0.23 and 0.40 for SFT on LLaMA-70B answers followed by GRPO (A+GRPO), and 0.30 and 0.40 for standalone GRPO. This suggests that self-revision training provides a better initialization for subsequent RL than either off-policy teacher data or training from scratch, consistent with the idea that the revision capability establishes useful reasoning patterns that RL can further refine. Notably, this experiment uses unfiltered self-revision traces, so the gains are likely a lower bound on what filtered self-revision (as used in SD-ZERO) would achieve.

Critical Assessment

Claim 1: SD-ZERO improves performance by at least 10% over base models

This claim is supported by the reported numbers but requires careful interpretation. What "10%" means: across the 8-benchmark average, Qwen3-4B-Instruct improves from 49.8% to 60.3% (+10.5 percentage points in absolute accuracy) and Olmo-3-7B-Instruct from 41.1% to 51.5% (+10.4 points). These are absolute accuracy differences, not relative improvements. The relative improvement is approximately 21% for Qwen3-4B and 25% for Olmo-3-7B, which is substantial but should not be confused with the absolute 10% figure.

The result genuinely holds across all 8 benchmarks for Qwen3-4B (no benchmark regresses) and across 7 of 8 for Olmo-3-7B (Codeforces drops slightly from 32.4% to 57.8%? — a careful reading shows this is actually a large improvement, not a drop; the base model's 32.4% on Codeforces rises to 57.8% for SD-ZERO, a +25.4 point gain). The per-benchmark improvements range from +2.0 points (Math for Olmo-3-7B) to +25.4 points (Codeforces for Olmo-3-7B), showing that the average gain is driven by large improvements on specific benchmarks rather than uniform uplift.

A limitation: the paper reports only accuracy (avg@8 and pass@8) and token counts. Metrics like ROUGE-L between generated and reference solutions, diversity of solution approaches (do all 8 samples produce the same correct answer or different valid approaches?), or calibration of the model's confidence are not reported. The claim of improvement is therefore narrowly about correctness on benchmarks with verifiable answers — it says nothing about whether the improved model produces more diverse, creative, or well-explained solutions.

Claim 2: SD-ZERO outperforms all baselines (RFT, GRPO, SDFT) under matched sample budgets

Supported with a specific qualification. The paper carefully matches total generation budgets (49K for SD-ZERO vs. 60K for baselines) and provides detailed accounting in Appendix C.2. SD-ZERO outperforms all baselines on the 8-benchmark average for both model families (Table 1). The GRPO comparison is additionally strengthened by Table 10 showing that even 2× the sampling budget does not close the gap.

The qualification: the baselines are run for a single training epoch on 15K questions. The paper acknowledges this in Section 3.3: "While we note that GRPO may benefit from additional training epochs, our comparison is under a matched single-epoch budget, where SD-ZERO achieves stronger performance with comparable total sample cost." This is a fair constraint for a matched-budget comparison, but it means the claim is specifically about sample efficiency under fixed data quantity, not about the ultimate ceiling of each method given unlimited training. A GRPO run for multiple epochs on the same 15K questions might eventually surpass single-epoch SD-ZERO, though Table 10's finding that doubling GRPO's sampling budget yields only +0.6 points suggests diminishing returns.

Additionally, the SFT baseline uses DeepSeek-R1 demonstrations — a specific external teacher. Different teacher choices (GPT-4, Claude, human-written solutions) might yield different SFT results. The paper's finding that SFT on DeepSeek-R1 traces degrades performance on AMOBench and LiveCodeBench for Qwen3-4B (Table 1) suggests distribution mismatch between the teacher traces and the student model, but this is one teacher — it's possible that SFT on a better-matched teacher (e.g., Qwen3-4B's own best outputs) would perform better. The RFT baseline is essentially this: SFT on the model's own best outputs. RFT performs substantially better than off-policy SFT, validating the importance of on-policy data, but still below SRT.

Claim 3: The reviser transforms binary rewards into dense token-level supervision with token-level self-localization

The evidence for this claim is strong and multi-faceted. Figure 4 provides the direct mechanistic evidence: per-token KL divergence is concentrated on a small fraction of tokens for incorrect responses but uniform for correct responses. The case study in Figure 4 (right) shows that tokens corresponding to a faulty symmetry argument receive large positive KL while tokens for a correct coordinate-based approach receive large negative KL — the teacher is providing a two-sided signal (penalize error tokens, redirect to correct tokens). This is exactly what "dense token-level supervision" would look like.

However, the paper does not quantify the relationship between token-level KL and actual error locations. Do the high-KL tokens consistently correspond to reasoning steps that are objectively wrong? The case study is illustrative but the paper does not provide a systematic study across many examples with human annotation of which tokens are erroneous. The claim that the reviser "identifies which tokens to correct" is based on qualitative inspection (the Figure 4 example) and the aggregate distribution pattern (Figure 4 left) rather than a quantitative evaluation against ground-truth error localization.

A stronger test would be: take a set of incorrect student responses, have human annotators mark which tokens are wrong, and measure whether the reviser's high-KL tokens overlap with the human-labeled errors. This experiment is not performed.

Additionally, the mechanism relies on the reviser having been trained on self-revision traces in Phase 1. Without SRT, the teacher produces essentially random token-level feedback (Table 12 shows the correction rate stays at 2.6%). This means the "dense token-level supervision" is a learned capability, not an emergent property of any model conditioned on its own responses. The claim generalizes only to models that have been explicitly trained to self-revise.

Claim 4: SD-ZERO achieves iterative self-evolution through teacher synchronization

Supported but demonstrated for only one additional round. Figure 5 shows that the first Self-Distillation phase saturates around step 400, and a second phase after teacher synchronization yields "at least 3 additional percentage points without signs of saturation." This is evidence for a second round of improvement, but the claim of "iterative self-evolution" implies that this process can continue over multiple rounds. The paper demonstrates only two rounds (SRT → Self-Distillation round 1 → teacher sync → Self-Distillation round 2). Whether a third, fourth, or fifth round would continue to yield gains without saturation is an open question.

The paper does not report what happens if teacher synchronization is performed more frequently (e.g., every 100 steps rather than after a full epoch). Does more frequent synchronization lead to faster improvement or instability? The experiment demonstrating only one synchronization point leaves this unexplored. The paper also does not explore whether the improvement plateaus after multiple rounds — the 3% gain from the second round is smaller than the 7.8% gain from SRT, suggesting diminishing returns that might asymptote.

The experiment is also performed only on Qwen3-4B-Instruct with OpenR1-Math. Whether iterative self-evolution works on Olmo-3-7B-Instruct, on code tasks, or on different data distributions is not tested. The paper's framing of this as a general capability of SD-ZERO ("Once primed by SRT, the model can continue to self-evolve through iterated teacher synchronization") is broader than the evidence supports — it is demonstrated in one specific setting for one additional round.

Strengths of the Experimental Design

Comprehensive baseline comparison under matched budgets. The paper goes beyond simply claiming superiority by carefully matching generation budgets (Appendix C.2, Table 3) and even exploring expanded budgets for GRPO (Table 10). This level of budget accounting is rare in post-training papers and makes the comparison substantially more credible.

Dual model evaluation. Testing on two different model families (Qwen3-4B, Olmo-3-7B) with different architectures and pretraining recipes reduces the risk that the results are specific to one model's idiosyncrasies. The consistent advantage of SRT and SD-ZERO across both families strengthens the generalizability claim.

Eight benchmarks spanning math and code, in-distribution and out-of-distribution. The evaluation suite covers competition math (AIME24, AIME25, HMMT25), olympiad math (AMOBench), general math (MATH, OpenR1-Math), and competitive programming (Codeforces, LiveCodeBench). The consistent improvements across diverse benchmarks (with the exception of some baseline-specific regressions like SFT on AMOBench) suggest that SD-ZERO's benefits are not benchmark-specific.

Multiple ablation studies testing boundary conditions. The ablated variants (no SRT phase, single loss term, unfiltered revisions, different data splits) test specific predictions of the method's design logic. The failures of these ablated variants (unsurprisingly) confirm the necessity of each component, but also provide insight into why they are necessary — e.g., Table 12 shows that without SRT, the teacher provides no useful feedback, confirming that the reviser's capability must be explicitly trained.

Negative results reported transparently. The paper includes experiments where things don't work: SDFT with thinking models degrades performance (Table 14), unfiltered self-revision data hurts (Table 15), applying Self-Distillation without SRT fails (Table 12), and ReST^EM-style optimization backfires (Appendix K, mentioned in the related work discussion). These negative results are informative and lend credibility to the positive claims by showing that the method is not trivially effective.

Weaknesses and Missing Experiments

Single-epoch training for all methods. As noted above, the comparison is under a matched single-epoch budget. While this is fair for sample-efficiency claims, it leaves open the question of whether GRPO (or SDFT, or RFT) with multi-epoch training on the same data would eventually match or exceed SD-ZERO's performance. The paper's response is that Table 10 partially addresses this (doubling GRPO's budget and training to a full epoch yields minimal improvement), but a systematic study of multi-epoch training across all baselines would strengthen the claim that SD-ZERO's advantage is structural rather than just a matter of optimal hyperparameter settings.

No confidence intervals or error bars. The paper reports exact accuracies without standard deviations, confidence intervals, or statistical tests. With a 500-question test set for in-distribution evaluation (and smaller for some out-of-distribution benchmarks), differences of 1–2 percentage points may not be statistically significant. The lack of uncertainty quantification makes it difficult to assess whether SD-ZERO's 2.7-point gain over SRT (60.3% vs. 57.6%) on Qwen3-4B-Instruct is reliably different from zero, or whether the 3-point gain from teacher synchronization (Figure 5) is within sampling noise.

No evaluation on non-reasoning benchmarks. All 8 benchmarks are math and code tasks with verifiable answers. The paper does not test whether SD-ZERO's improvements transfer to general language understanding (MMLU), instruction following (IFEval), or other domains. This is by design — the method requires binary verifiability — but it means the claim of "at least 10% improvement" is domain-specific. The Countdown experiments in Appendix G use a constrained arithmetic task, still within the verifiable reasoning domain, so they do not expand the domain coverage.

Limited scale exploration. All experiments use 4B and 7B parameter models. Whether SD-ZERO's advantages persist, diminish, or amplify at larger scales (e.g., 70B, 405B) is unknown. Larger models may have stronger base revision capabilities (making SRT less necessary) or may benefit differently from the Self-Distillation phase. The paper does not discuss scaling behavior or provide even a single experiment at a different scale.

No comparison to process reward model (PRM) training. The paper argues (Appendix A) that the reviser's token-level signal is "functionally analogous to process reward models." However, no experiment compares SD-ZERO against a model trained with a PRM — e.g., using best-of-N weighted selection with a PRM verifier, or using PRM-guided search at inference. Given that PRMs are the standard approach for converting sparse rewards into dense signal, a direct comparison would substantially strengthen the claim that the reviser provides comparable or superior supervision.

The teacher synchronization experiment is minimal. Only one synchronization point is tested, on one model, on one dataset. The 3% gain is reported without error bars. The claim that this demonstrates "iterative self-evolution" implies a general capability that the evidence supports only weakly. A minimal additional experiment — a third synchronization round or replication on Olmo-3-7B — would substantially strengthen this claim.

No analysis of solution diversity or mode collapse. The paper reports Pass@8 improvements (Table 7) and interprets them as evidence that SD-ZERO doesn't just sharpen the output distribution. However, Pass@8 is a coarse metric — the model could be producing 8 identical correct solutions (rather than 8 diverse correct approaches) and still achieve high Pass@8. Metrics like the number of distinct correct solutions, semantic diversity of solution approaches, or the entropy of the output distribution across samples are not reported. The concern about distribution collapse is specifically relevant given that distillation is known to reduce output diversity in some settings.

Missing baseline: SD-ZERO with an external teacher. A natural experiment to understand the value of self-revision specifically would be to compare SD-ZERO (self-teacher) against an otherwise identical pipeline that uses an external teacher model (e.g., a larger Qwen variant) for the Self-Distillation phase. This would isolate whether the gains come from the distillation format or specifically from the self-revision capability. The paper does not run this experiment.

The ablation on data splits (Table 13) has small performance differences. The three data splits produce SRT accuracies of 57.6%, 57.8%, 57.8% and SD-ZERO accuracies of 60.3%, 59.1%, 59.8%. The 1.2-point gap between the best and worst SD-ZERO configurations (60.3% vs. 59.1%) is small and may not be statistically significant. The paper's recommendation to allocate more data to Phase 2 is sensible given the direction of the results, but the evidence for it is not strong.

Conditions Under Which the Claims Hold

Based on the experimental evidence:

1. SD-ZERO's improvement over baselines holds under matched single-epoch, matched-generation-budget conditions on Qwen3-4B-Instruct and Olmo-3-7B-Instruct for math and code reasoning benchmarks. The claim explicitly does not extend to multi-epoch training, larger models, or non-reasoning domains.

2. The reviser's token-level self-localization holds for the SRT-trained reviser, as demonstrated qualitatively in Figure 4 and quantitatively in the aggregate KL distribution. The claim does not extend to base models without SRT (as shown in Table 12) or to other models with different revision training procedures.

3. The Self-Distillation phase compresses response length by approximately 2× while maintaining or improving accuracy, as demonstrated in Figure 3 and Table 9. This claim holds for Qwen3-4B-Instruct on AIME24; the paper does not provide per-benchmark response length data to confirm it generalizes across all benchmarks.

4. Iterative teacher synchronization yields at least 3 additional percentage points for one additional round on Qwen3-4B-Instruct with OpenR1-Math. The claim of open-ended "iterative self-evolution" is speculative and not demonstrated beyond two rounds.

5. SD-ZERO requires only binary outcome supervision, as demonstrated by the entire pipeline's design and confirmed by the SDFT-with-final-answers-only ablation (Table 8). This is the most robust claim in the paper, as it is a design property rather than an empirical result.

6. Limitations and Trade-offs

Verification Dependency: The Entire Pipeline Requires Binary Outcome Verifiability

The assumption or constraint. SD-ZERO fundamentally depends on the availability of a binary verifier $r(y, a) \in \{0, 1\}$ that can determine whether a generated response's final answer matches the ground-truth answer. The paper is explicit about this scope: "We focus on verifiable domains such as math and coding" (Section 5), and acknowledges that "Extending SD-ZERO to domains without verifiable rewards remains an open problem" (Section 5).

This is not a minor implementation detail — it is a structural requirement at every stage of the pipeline. Phase 1 (SRT) requires the binary verifier to (a) determine which initial responses are correct vs. incorrect so the appropriate control prompt can be constructed, and (b) filter revision traces to retain only those where the revised response is correct. Phase 2 (Self-Distillation) requires the binary verifier at every training step: the student generates a response, the verifier checks it, and the resulting binary reward determines which control prompt the reviser teacher receives, which in turn shapes the token-level KL targets. Without this signal, the teacher cannot know whether to condition on "rephrase" or "start over," and the entire mechanism for converting sparse reward into dense supervision collapses.

The consequence. SD-ZERO is inapplicable to a vast range of tasks where binary correctness cannot be automatically determined. This includes: open-ended generation (creative writing, dialogue, summarization), tasks where correctness is multi-dimensional or subjective (essay grading, policy analysis), tasks where ground-truth answers exist but automatic extraction and matching is unreliable (complex structured outputs, code where functional correctness requires execution), and tasks where the "answer" is a plan or recommendation that cannot be evaluated without environment interaction. The paper's Countdown experiments (Appendix G) remain within the verifiable arithmetic domain and do not expand applicability.

Even within verifiable domains, the binary verifier introduces a subtlety: the verifier checks only the final answer, not the reasoning. A model can produce a correct final answer through faulty reasoning (false positives for the verifier) or an incorrect final answer despite largely sound reasoning with a single arithmetic slip (false negatives). SRT's filtering step keeps revision traces where the revised response is correct, but the initial response may have been marked incorrect despite containing useful partial reasoning. RFT discards such responses entirely; SRT preserves them as context for the revision, which is an improvement, but the binary reward still coarsely labels the entire reasoning chain. The paper does not investigate whether this coarseness introduces noise into the training signal — for instance, whether the reviser sometimes "corrects" reasoning that was actually sound but had a final-answer extraction error.

What evidence exists in the paper. The paper's entire evaluation suite (Table 1) consists exclusively of math and code benchmarks with verifiable answers: competition math (AIME24, AIME25, HMMT25), olympiad math (AMOBench), general math (MATH, OpenR1-Math), and competitive programming (Codeforces, LiveCodeBench). There is no experiment, even a small-scale one, on any non-verifiable task. The limitation is therefore demonstrated by omission — the paper provides no evidence that SD-ZERO's mechanism could function without binary verifiability.

Mitigation status. The paper acknowledges this limitation in Section 5 and suggests one speculative direction: "One promising direction is to define rewards using meta-cognitive signals (Didolkar et al., 2024; 2025; Shao et al., 2025), such as consistency or self-correction." This is a pointer to future work, not a mitigation. No experiment tests whether auto-generated binary signals (e.g., from LLM-as-judge, self-consistency voting, or constraint satisfaction checks) could substitute for ground-truth verifiers. The Countdown experiments in Appendix G still use ground-truth verification. The limitation is therefore entirely unaddressed in the current work, and a practitioner looking to apply SD-ZERO to tasks outside math/code would need to solve the verification problem independently.

---

Thinking Models Are Explicitly Out of Scope, Yet Represent the Most Important Modern Deployment Regime

The assumption or constraint. The paper limits its scope to "instruct models that generate short and concise responses" (Section 5) with thinking mode disabled during training ("Thinking: False" in Table 4). This is a deliberate choice motivated by a specific failure mode: when SDFT is applied to Qwen3-4B with thinking enabled during the student's on-policy rollouts, performance degrades substantially on competition math benchmarks (Table 14). The paper explains:

"extending self-distillation to thinking models, which produce long, exploratory chains of thought... is challenging because such responses may include false starts and partial corrections that are not mistakes, making it difficult to distinguish productive exploration from genuine errors and assign credit beyond local token decisions" (Section 5).

The core problem is that thinking models naturally generate reasoning that contains backtracking, reconsideration, and exploration — tokens that look like errors or corrections but are actually productive parts of the reasoning process. The reviser teacher, trained on clean outcome-conditioned revision traces, cannot distinguish between a "false start" that the thinking model deliberately explores and a genuine mistake that should be corrected. The teacher's token-level KL signal would therefore penalize tokens that are actually useful, potentially degrading the model's reasoning rather than improving it.

The consequence. SD-ZERO is incompatible with the most important category of modern reasoning models. The models that achieve state-of-the-art results on the paper's own benchmarks — DeepSeek-R1, OpenAI o1/o3, Gemini thinking models, Qwen3 with thinking enabled — all use extended chain-of-thought reasoning with explicit exploration, backtracking, and verification. These are precisely the models that produce the "false starts and partial corrections" that SD-ZERO's reviser cannot handle. A practitioner looking to improve a thinking model's reasoning would find SD-ZERO inapplicable without substantial modification to the revision training procedure and the teacher's conditioning mechanism.

Furthermore, this limitation fundamentally constrains the ceiling of SD-ZERO's approach. The paper trains non-thinking models (Qwen3-4B-Instruct, Olmo-3-7B-Instruct) and achieves 10%+ gains, but these gains are relative to non-thinking baselines. A thinking-enabled version of the same base model would start from a substantially higher baseline (as Table 14 shows: Qwen3-4B with thinking achieves 73.5% on AIME24 vs. 59.6% for the non-thinking instruct variant used in the main experiments). SD-ZERO's 68.3% on AIME24, while impressive relative to the non-thinking base model's 59.6%, is still below what the model achieves by simply enabling thinking. The method improves non-thinking reasoning but does not provide a path to matching or exceeding thinking-model performance.

What evidence exists in the paper. Table 14 is the key evidence: applying SDFT (the closest existing method to SD-ZERO's Phase 2) to Qwen3-4B with thinking enabled during training causes accuracy drops of 9.8 points on AIME24, 10.8 points on AIME25, and 8.3 points on HMMT25 relative to the base thinking model. Disabling thinking during training largely preserves the base model's accuracy but does not improve it. The paper does not run an equivalent experiment for SD-ZERO specifically (SRT + Self-Distillation with thinking enabled), so the degradation of SD-ZERO on thinking models is inferred from SDFT's behavior rather than directly measured. However, since SDFT and SD-ZERO share the same Phase 2 mechanism (on-policy self-distillation with a conditional teacher), the failure mode is likely transferable.

Mitigation status. The paper explicitly defers this to future work in Section 5: "An important next step is extending self-distillation to thinking models." No mitigation is attempted. The paper does not propose modifications to the SRT data collection (e.g., training the reviser on thinking-model traces that include productive backtracking), to the teacher's conditioning (e.g., providing the teacher with segment-level rather than token-level feedback), or to the distillation objective (e.g., masking tokens that correspond to exploration rather than errors). This is identified as an open problem with preliminary negative evidence suggesting it is non-trivial.

The practical implication is that a practitioner choosing between SD-ZERO and simply enabling thinking mode on their model faces a direct tradeoff: SD-ZERO can improve a non-thinking model's accuracy by 10%+ while keeping responses concise (~3500 tokens), but the resulting model may still underperform the same base model with thinking enabled (~73.5% on AIME24 with ~8000+ tokens). Whether the accuracy gap is worth the token savings depends on the deployment context, but the paper does not provide this comparison.

---

The SRT Data Collection Cost Is Only Partially Amortized in the Headline Training Budget

The assumption or constraint. The paper's training budget accounting (Table 3, Appendix C.2) includes all generations used for SRT data collection — 10K initial responses + 30K revision attempts = 40K generations — and compares these against baselines that use 60K generations (RFT, GRPO, SDFT). SD-ZERO's 49K total generations is presented as a sample-efficiency advantage. However, this accounting omits a critical cost: the revision success rate during SRT data collection is low, and the paper does not amortize the cost of failed revision attempts into the usable training data.

Specifically, Appendix C.1 reports that from 10K seed questions, the SRT pipeline produces only 6K usable self-revision traces. This means 30K revision attempts (5K correct initial responses × 3 rephrases + 5K incorrect × 3 corrections) yielded only 6K correct revisions — a 20% success rate. The 24K failed revision attempts are generated, verified, and discarded. Their computation cost is included in the 40K generation budget, but the resulting training data is only 6K traces. By contrast, RFT's 60K generations are filtered to some number of correct responses (depending on the base model's pass@1), so RFT also discards generations, but the filtering rate is not reported. The paper does not compare the "generations per usable training example" ratio across methods.

The consequence. The effective sample efficiency of SD-ZERO depends heavily on the base model's revision capability before SRT. If the base model has a low revision success rate (as the paper reports: "our base models exhibit strong generator performance but weak reviser performance"), SRT data collection requires many revision attempts per usable trace. The paper's 20% success rate (6K traces from 30K attempts) means each SRT training example costs ~5 revision generations. If a different base model had, say, a 10% success rate, SRT would require ~10 attempts per trace, substantially increasing the Phase 1 generation budget.

This also creates a practical calibration problem: a practitioner cannot know in advance how many seed questions and revision attempts will be needed to produce a target number of SRT training traces. The paper uses 10K seed questions and 3 revision attempts per initial response to get 6K traces, but these numbers depend on the base model's accuracy (which determines the correct/incorrect split of initial responses) and its revision capability (which determines the success rate of revision attempts). A weaker base model would need more questions, more revision attempts, or both, increasing the Phase 1 cost beyond what the paper's budget accounting assumes.

What evidence exists in the paper. The numbers are provided in Appendix C.1: "10K initial responses are roughly split into 5K correct and 5K incorrect responses," "For each correct initial response, prompt the model to generate 3 rephrased responses; For each incorrect initial response, prompt the model to generate 3 corrected responses," and the result is "6K self-revision traces." The 20% success rate is not explicitly stated but is computed from these numbers: 30K attempts → 6K successes. The paper does not report revision success rates for different difficulty levels, for Olmo-3-7B-Instruct, or for code tasks, so the generalizability of this 20% figure is unknown.

Mitigation status. The paper does not address this cost directly. The generation budget comparison in Table 3 presents 40K SRT generations as a one-time cost, without analyzing how the usable-traces-per-generation ratio affects the comparison. The paper does not explore whether fewer revision attempts per question (e.g., 2 instead of 3) would produce nearly as many usable traces at lower cost, or whether more attempts (e.g., 5) would yield enough additional traces to justify the cost. The paper does not discuss whether the 24K failed revisions could be used for other purposes (e.g., as negative examples in a contrastive training setup) to amortize their cost. This is a practical consideration for anyone replicating SD-ZERO, since the Phase 1 cost must be paid upfront before any training benefit is realized, and the cost scales with the number of revision attempts per question.

---

Single-Epoch Training and Absence of Statistical Rigor Leave the Upper Bound and Reliability Unclear

The assumption or constraint. All comparisons in the paper are conducted under a matched single-epoch training budget. The paper states this explicitly for GRPO: "our comparison is under a matched single-epoch budget" (Section 3.3). The GRPO baseline is run for 1 epoch on 15K questions with 4 rollouts per question. SFT, RFT, and SDFT are also run for 1 epoch (SFT/RFT for 3 epochs on a smaller effective dataset, as specified in Table 4, but the total optimization steps are comparable). SD-ZERO itself uses 1 epoch of SRT (3 epochs on the 6K SRT traces per Table 4) and 1 epoch of Self-Distillation.

The consequence of this design choice is that the paper measures sample efficiency at a fixed data quantity, not the asymptotic performance ceiling of each method. GRPO, in particular, is known to benefit from multiple epochs of training on the same data — each epoch generates new on-policy rollouts, and the policy improvement from earlier epochs can compound. The paper's Table 10 partially addresses this by testing GRPO with 8 generations per question at 1 epoch, which provides only a +0.6 point improvement over 4 generations at 1 epoch, but this still tests only one epoch of training. It does not test GRPO trained for 2, 3, or 5 epochs on the same 15K questions, which would allow the policy to improve iteratively through multiple rounds of on-policy data collection and advantage estimation.

Furthermore, the paper reports all results as point estimates without confidence intervals, standard deviations, or statistical tests. For a 500-question test set (for in-distribution OpenR1-Math and Codeforces), a difference of 1-2 percentage points represents 5-10 questions — easily within the range of sampling variability. The 2.7-point gap between SD-ZERO and SRT on Qwen3-4B-Instruct (60.3% vs. 57.6%) could reflect genuine improvement or could partially reflect noise, and the paper provides no way to distinguish these possibilities. The 3-point gain from teacher synchronization in Figure 5 is reported on an unspecified evaluation set size with no error bars, making it impossible to assess whether the trend is reliable or could be explained by variance.

What evidence exists in the paper. Table 10 demonstrates that simply increasing GRPO's rollout budget within a single epoch yields minimal gains, which partially addresses the concern about within-epoch sample allocation. However, the experiment does not test multi-epoch GRPO, which could discover better reasoning strategies through iterative policy improvement that are not accessible from a single epoch's worth of gradient updates regardless of how many rollouts are used. The distinction is between "more samples per update" (tested in Table 10) and "more updates with fresh on-policy samples" (not tested).

For the statistical reliability concern, the paper provides no uncertainty quantification anywhere. The 500-question test sets are split into difficulty bins in some analyses, and the iterative self-evolution experiment (Figure 5) presumably uses a subset, but sample sizes are not reported alongside the accuracy numbers. The lack of error bars on all figures (Figures 3, 5, 6) and the reporting of exact percentages without confidence intervals throughout the paper make it impossible to assess which reported differences are statistically meaningful.

Mitigation status. The multi-epoch limitation is partially acknowledged in Section 3.3: "While we note that GRPO may benefit from additional training epochs, our comparison is under a matched single-epoch budget." Table 10 provides some evidence that increased within-epoch sampling does not close the gap, but this is not equivalent to testing multi-epoch training. The paper does not run any multi-epoch baseline.

The statistical reliability limitation is entirely unaddressed. No confidence intervals, standard deviations, or statistical tests are reported for any result. This is a significant weakness for a paper making quantitative claims about method superiority, particularly when some of the claimed advantages (2.7 points for Phase 2 over SRT, 3 points for teacher synchronization) are small enough that sampling variance could be a meaningful fraction of the reported effect. Replication across two model families (Qwen3-4B and Olmo-3-7B) provides some robustness, but consistent direction of effect does not substitute for within-experiment uncertainty quantification.

---

The Revision Capability Is Not Evaluated Against Direct Alternatives for Providing Dense Supervision

The assumption or constraint. The paper's central claim is that the SRT-trained reviser transforms sparse binary rewards into dense token-level supervision, and that this supervision is more effective than the sparse signal used by RLVR methods (GRPO) or the privileged-information signal used by self-distillation methods (SDFT). The paper compares against GRPO (which uses only binary rewards) and SDFT (which uses gold solution traces as privileged context), but does not compare against methods that produce dense token-level supervision through other mechanisms — most notably, process reward models (PRMs).

PRMs (Lightman et al., 2024; Wang et al., 2024) are the standard approach for converting sparse outcome rewards into dense step-level feedback. A PRM trained on the base model's outputs (e.g., via Monte Carlo rollout supervision as in Wang et al., 2024) could provide per-step correctness probabilities during training, enabling token-level or step-level advantage estimation that is functionally analogous to what the reviser provides through KL divergence. The paper explicitly notes this connection in Appendix A:

"The per-token KL signal from the reviser is functionally analogous to process reward models (PRMs)... in that both provide localized supervision over intermediate reasoning rather than only the final outcome."

Given this acknowledged analogy, a comparison between SD-ZERO and PRM-based training would directly test whether the reviser's implicit token-level feedback is competitive with explicit PRM-trained feedback — and whether the SRT phase's cost (40K generations for data collection) is justified relative to PRM training cost.

The consequence. Without a PRM comparison, the paper cannot distinguish between two interpretations of SD-ZERO's gains: (a) the reviser provides uniquely effective token-level supervision because it is conditioned on the student's specific errors, or (b) any form of dense token-level or step-level supervision substantially outperforms sparse binary rewards, and SD-ZERO is one way (but possibly not the most efficient way) to obtain such supervision.

If interpretation (b) is correct, then a practitioner might achieve comparable or better results by training a PRM (which requires no SRT phase, only rollout-based supervision data) and using it for either RL with dense rewards or best-of-N reranking. The cost of training a PRM would need to be compared to the cost of SRT data collection (40K generations) plus Self-Distillation training. The paper provides no data to inform this comparison.

Furthermore, the lack of a PRM comparison makes it difficult to assess whether the reviser's "token-level self-localization" property (Figure 4) is genuinely informative or merely reflects the reviser's training distribution. A PRM's per-step scores would provide an independent measure of which reasoning steps are likely correct, which could be compared against the reviser's high-KL tokens to validate that the reviser is identifying genuine errors rather than producing plausible-sounding but arbitrary corrections.

What evidence exists in the paper. No PRM is trained, and no comparison against PRM-based methods is performed. The paper's baselines are RLVR (GRPO), supervised fine-tuning (SFT, RFT), and self-distillation (SDFT). The PRM connection is mentioned only in Appendix A as a conceptual analogy, not as an empirical comparison point.

Mitigation status. The paper does not address this gap. The PRM analogy in Appendix A serves to position SD-ZERO within the broader literature but does not motivate an experiment that would test whether the reviser's implicit feedback is competitive with explicit PRM feedback. The paper suggests that "whether this implicit token-level signal can complement or serve as a cheaper alternative to trained PRMs in settings such as guided tree search remains an interesting direction for future work," but the comparison is deferred entirely. A practitioner choosing between implementing SD-ZERO and training a PRM has no evidence from this paper to guide the decision.

---

The Teacher Synchronization Claim Overpromises Relative to the Evidence

The assumption or constraint. The paper presents "iterative self-evolution" through teacher synchronization as a key capability of SD-ZERO, formalized in Section 3.4 and Figure 5. The framing suggests that the model can continue improving through repeated rounds of Self-Distillation with synchronized teachers:

"Once primed by SRT, the model can continue to self-evolve through iterated teacher synchronization, requiring only the initial attempt and corresponding binary reward in the teacher's context." (Section 3.4)

The consequence. This claim substantially overstates what the evidence supports. Figure 5 shows exactly one teacher synchronization: the first Self-Distillation phase saturates around step 400 at approximately 57% accuracy, the teacher is synchronized with the updated student, and a second phase yields "at least 3 additional percentage points without signs of saturation" — reaching approximately 60% accuracy and still improving at the end of the plotted training steps.

A single synchronization event does not demonstrate iterative self-evolution — it demonstrates a two-stage improvement (SRT → Self-Distillation round 1 → teacher sync → Self-Distillation round 2), which is different from the open-ended iterative process the paper's language implies. Several critical questions are unanswered:

• Would a third synchronization round yield further gains, or does the process saturate after two rounds?
• Does the gain per round diminish? (The first Self-Distillation phase provides +2.7 points over SRT on the full benchmark suite; the synchronization provides ~3 points on OpenR1-Math specifically. Are these comparable magnitudes, and can they be sustained?)
• What is the optimal synchronization frequency? Would synchronizing more often (e.g., every 100 steps) yield faster or less stable improvement?
• Does the process work on Olmo-3-7B-Instruct, on code tasks, or on different data distributions? Figure 5 tests only Qwen3-4B-Instruct on OpenR1-Math.

The paper's language in Section 3.4 ("the model can continue to self-evolve," "the loop to sustain") extrapolates from one synchronization event to a general capability, which is not justified by the presented evidence.

What evidence exists in the paper. Figure 5 is the sole evidence for iterative self-evolution. It shows training curves for a single model (Qwen3-4B-Instruct) on a single dataset (OpenR1-Math) with one teacher synchronization event. The y-axis shows "Accuracy (%)" but the evaluation set and sample size are not specified, and no error bars are provided. The 3-point gain is visually apparent but not statistically quantified.

Mitigation status. No additional synchronization rounds are tested. No ablations on synchronization frequency are performed. No replication on Olmo-3-7B-Instruct or on code tasks is provided. The paper presents the result as evidence of a general capability but acknowledges none of these limitations in the discussion. Section 5 ("Limitations and Open Directions") focuses on the thinking-model limitation and the verifiability requirement, not on the limited evidence for iterative self-evolution. The claim therefore stands as the weakest empirical link in the paper's argument — it is a promising result that warrants the limited framing of a preliminary finding ("SD-ZERO can be extended through teacher synchronization, with a second round yielding additional gains") rather than the stronger claim of iterative self-evolution that the paper advances.

7. Implications and Future Directions

How This Work Changes the Landscape

SD-ZERO introduces a conceptual reframing that shifts how we think about supervision in post-training for verifiable domains. The prevailing taxonomy has organized methods along an axis of supervision density: at one end, RLVR methods like GRPO accept sparse binary rewards and pay for it in sample inefficiency; at the other end, distillation methods demand dense token-level supervision from external teachers or gold solution traces. SD-ZERO breaks this dichotomy by demonstrating that a single model can generate its own dense token-level supervision from sparse binary rewards, provided it has first been trained to self-revise. This is not an incremental improvement along the density axis — it creates a new category where the supervision is self-generated rather than externally provided or accepted as sparse.

The magnitude is a reframing rather than a paradigm shift. SD-ZERO does not make RLVR obsolete (it still requires binary verifiability and is untested on thinking models), nor does it displace distillation from external teachers (stronger teachers likely still provide better supervision when available). What it does is open a third path that was previously unavailable: if you have a verifiable domain and a base model with non-trivial generation accuracy — even without expert demonstrations — you can bootstrap to substantially stronger performance using only the model's own outputs and a binary reward signal. The paper's consistent 10%+ absolute accuracy gains over base models (49.8% → 60.3% for Qwen3-4B-Instruct, 41.1% → 51.5% for Olmo-3-7B-Instruct, Table 1) demonstrate that this path is not merely theoretically possible but practically competitive with or superior to existing methods under matched training sample budgets.

The work reconciles a tension that has been building in the self-correction literature. Earlier studies (Huang et al., 2023) found that "large language models cannot self-correct reasoning yet" — base models prompted to revise their own outputs showed negligible improvement, consistent with this paper's finding that the base Qwen3-4B-Instruct gains only 1.1% from revision (Figure 3, Table 9). Yet more recent work on training-based self-correction (Kumar et al., 2024b) showed that RL can teach models to correct errors. SD-ZERO provides a mechanistic explanation for the discrepancy: base models lack the outcome-conditioned revision capability. They can generate responses and they can generate corrections when shown the correct answer, but they cannot condition on their own incorrect attempt and a binary reward to produce a targeted correction. SRT teaches precisely this capability, and once acquired, the model's correction rate jumps from 2.7% to 15.0% (Table 9). The prior "cannot self-correct" finding was not about an inherent limitation of self-correction but about a specific capability gap in base models — one that SRT fills.

The paper also redirects research attention in two specific ways. First, it suggests that investing in revision capability is a high-leverage research direction. Prior work treated self-correction as either a prompting strategy (Madaan et al., 2023; Shinn et al., 2023) or a byproduct of RL training (Kumar et al., 2024b). SD-ZERO shows that targeted training on error-to-correction trajectories — even with modest data (6K traces) — produces a revision capability that can then be leveraged as a supervision generator for further training. This makes revision capability not just a useful inference-time skill but a training-time resource: the better the model can revise, the richer the token-level feedback it can provide to itself during distillation. This reframes revision from a downstream benefit (the model can fix mistakes at inference) to an upstream enabler (the model can teach itself to make fewer mistakes during training).

Second, the work redirects attention away from developing ever-more-sophisticated RL credit assignment mechanisms and toward conditioning architectures for self-supervision. GRPO's limitation, as documented in Table 10, is not primarily that it needs better advantage estimators — doubling the rollout budget from 4 to 8 generations yields only +0.6 points on average (51.7% → 52.3%). The limitation is that binary rewards fundamentally provide too little information per sample, and no amount of clever credit assignment can recover information that isn't there. SD-ZERO's solution is architectural rather than algorithmic: condition the model on its own output and its binary reward, and let the model's learned revision capability unpack the sparse signal into dense token-level feedback. This suggests that future work on sample-efficient post-training should focus more on what the model conditions on and less on how gradients are computed from sparse signals.

The iterative self-evolution finding (Figure 5), while demonstrated for only one additional round, hints at a more profound shift: revision capability may be self-amplifying. If improving generation improves revision (because the same model does both), and improving revision provides better supervision for further generation improvements, then a model initialized with revision capability enters a virtuous cycle that standard self-training (which only uses correct generations for training) cannot replicate. Standard self-training is limited by the model's generation accuracy — it can only train on examples the model already gets right. SD-ZERO's Self-Distillation phase trains on all examples, using the reviser to extract useful supervision even from incorrect generations. This means the effective training signal per generated token is higher, potentially enabling faster or more sustained improvement curves. The paper provides only preliminary evidence for this (one synchronization round), but the conceptual mechanism is well-defined and testable.

Follow-Up Research This Work Enables

Quantitative evaluation of the reviser's token-level error localization against ground-truth error annotations. The paper's claim that the reviser achieves "token-level self-localization" (Section 4.1, Figure 4) is supported by qualitative case studies and aggregate KL distribution patterns, but not by systematic comparison against human-annotated error locations. A strong follow-up would: (1) collect a dataset of incorrect model responses on math problems with human annotations marking the first token where reasoning goes off-track, (2) compute the reviser's per-token KL divergence for these responses, and (3) measure whether high-KL tokens spatially correlate with human-labeled error locations. Metrics would include precision/recall of high-KL tokens against error spans, and whether the reviser's signal at the earliest error token is strong enough to be useful for early stopping or guided search. This experiment would distinguish between the reviser genuinely identifying errors versus producing a diffuse "this whole response is wrong" signal that happens to concentrate on some tokens by chance. The paper's current evidence (one case study in Figure 4 right) is suggestive but insufficient to validate the mechanism at scale.

SD-ZERO on thinking models: training the reviser to handle productive false starts. The paper identifies thinking models as a critical open challenge (Section 5, Table 14) and provides preliminary negative evidence that standard self-distillation degrades thinking-model performance. The core problem is that thinking models naturally generate backtracking and reconsideration tokens that look like errors but are actually productive. A direct follow-up would modify the SRT data collection: instead of training the reviser to correct errors in non-thinking responses, train it on thinking-model traces where the "error" is a false start and the "correction" is the subsequent productive reasoning. Concretely: sample thinking traces from a model with thinking enabled, segment them into (false start, recovery) pairs using heuristics based on explicit backtracking markers (e.g., "wait," "alternatively," "let me reconsider"), and train the reviser to generate the recovery segment when conditioned on the false start segment. The revised distillation objective would then mask tokens corresponding to productive exploration (rather than penalizing them via KL divergence), potentially using the reviser's ability to distinguish exploration from error as a gating mechanism. Success would be measured by whether a thinking model trained with this modified SD-ZERO maintains or improves its accuracy rather than degrading as in Table 14.

Comparison against process reward model (PRM) training under matched budgets. The paper acknowledges the functional analogy between the reviser's token-level signal and PRM step-level scores (Appendix A) but provides no empirical comparison. A direct follow-up would train a PRM on the same base model's outputs using Monte Carlo rollout supervision (Wang et al., 2024) with a generation budget matched to SD-ZERO's Phase 1 cost (~40K generations for SRT data collection). The PRM would then be used for either (a) best-of-N weighted selection at inference on the base model, or (b) dense-reward RL where the PRM provides per-step advantage estimates. The comparison would quantify: (1) the accuracy achieved by each approach under matched total generation budgets, (2) the per-sample training efficiency (gradient steps per usable training signal), and (3) whether the reviser's conditioning on specific student errors provides supervision that a generic PRM (trained on i.i.d. samples, not conditioned on particular error patterns) cannot match. The paper's iterative self-evolution finding (Figure 5) raises an additional question: can a PRM be iteratively refined as the model improves, and does the reviser's co-evolution with the generator (since they share parameters) provide an advantage over a separately trained PRM that must be retrained from scratch?

Multi-round iterative self-evolution with rigorous saturation analysis. The paper demonstrates one teacher synchronization (Figure 5) yielding ~3 percentage point gain, but frames this as evidence for "iterative self-evolution" — an open-ended capability. A systematic follow-up would run SD-ZERO for 5-10 synchronization rounds on a fixed dataset, measuring: (1) the accuracy gain per round and whether it follows a power-law or exponential decay, (2) whether revision capability (measured via Generate-then-Revise correction rate, as in Table 9) continues to improve alongside generation accuracy or saturates earlier, (3) the response length dynamics across rounds — does the distillation continue to compress responses further, or is most of the compression achieved in the first round, and (4) whether the process eventually overfits to the training distribution (measured by widening the gap between training and held-out performance). This experiment would determine whether "iterative self-evolution" is a genuine sustained improvement mechanism or a diminishing-returns effect that asymptotes after 2-3 rounds. It would also establish practical guidance on when to stop synchronization.

Cross-model self-distillation: does the reviser provide useful supervision to a different model? The paper uses the same model as both student and teacher, leveraging the fact that the reviser was trained on the model's own error patterns. An important extension is whether the reviser's token-level feedback generalizes to a different student model — for instance, training the reviser on Qwen3-4B's errors and using it to supervise Olmo-3-7B's training, or using a larger model's reviser to supervise a smaller model's distillation. This would test whether the revision capability captures general error-correction patterns (applicable across models) or is specific to the error distribution of the model it was trained on. A positive result would make SD-ZERO a general framework for teacher-student training where the teacher is a self-revision-trained model that can provide token-level feedback without needing gold solutions — enabling a new class of cross-model distillation that doesn't require the teacher to be strictly "smarter" than the student, only to have been trained on self-revision. The paper's Countdown experiments (Appendix G) show that off-policy data from stronger teachers underperforms on-policy self-revision data, suggesting distribution match matters more than teacher quality — a cross-model experiment would directly test this.

SD-ZERO as a replacement for the SFT stage in RLVR pipelines. The standard post-training recipe for reasoning models is SFT on high-quality demonstrations followed by RLVR (GRPO or similar). Table 1 shows that SFT on DeepSeek-R1 traces can degrade performance on some benchmarks (AMOBench: 9.8% → 7.3% for Qwen3-4B). SD-ZERO's Phase 1 (SRT) outperforms SFT on all benchmarks while using only 6K self-generated traces vs. 15K external demonstrations. A natural follow-up would test whether SRT can serve as a drop-in replacement for the SFT stage: train a model with SRT, then apply GRPO on top, and compare against the standard SFT→GRPO pipeline. The hypothesis is that SRT provides a better initialization for RL because it teaches the model to evaluate and correct its own reasoning, which may make the binary reward signal from GRPO more informative (the model can better interpret why a response was wrong and adjust accordingly). Appendix G.2 provides preliminary evidence for this direction: on the Countdown task, self-revised targets (A′) provide a better RL initialization than LLaMA-70B answers (A) for out-of-distribution benchmarks, with A′+GRPO achieving pass@128 of 0.40 on AIME24 vs. 0.23 for A+GRPO. Replicating this at scale on the main experimental setup and measuring whether SRT→GRPO outperforms SFT→GRPO under matched budgets would be a high-impact experiment with immediate practical implications.

Practical Applications and Downstream Use Cases

Data-efficient fine-tuning for domain-specific reasoning tasks. A team with a base instruct model and 5,000-15,000 question-answer pairs in a specialized verifiable domain (medical diagnosis from structured case data, legal statute interpretation with objective answers, physics problem-solving with numeric answers) can apply SD-ZERO without needing to collect expert reasoning traces. The paper's sampling budget analysis (Appendix C.2) provides a concrete recipe: start with ~10K seed questions, collect ~6K revision traces through the SRT pipeline (cost: ~40K model generations), train SRT, then run Self-Distillation on an additional ~9K questions (cost: ~9K on-policy generations). Total cost is ~49K generations, comparable to running GRPO on 15K questions with 4 rollouts (60K generations). The expected gain is approximately +10% absolute accuracy over the base model (based on Table 1 averages) without requiring any external teacher model or human-written reasoning demonstrations. This directly reduces the cost of domain adaptation for verifiable reasoning tasks where expert demonstrations would be expensive or unavailable.

On-device model improvement through self-play on verifiable tasks. For deployment scenarios where a smaller model (4B-7B parameters, as in the paper's experiments) runs locally and has access to a verifier (e.g., unit tests for code, symbolic math checkers, constraint solvers), SD-ZERO enables the model to improve itself on the specific distribution of problems it encounters. The pipeline is: (1) collect a buffer of questions the model answered incorrectly during deployment (along with their correct answers from the verifier), (2) periodically run SRT on these questions to update revision capability, (3) run Self-Distillation to compress improvements into the generator. Because SD-ZERO requires no external teacher, this improvement loop can run entirely on-device or on modest GPU infrastructure. The iterative self-evolution finding (Figure 5) suggests that multiple rounds can compound, meaning the model could continue improving over its deployment lifetime. The key practical constraint is the verifier: if the deployment domain has reliable automatic verification (as in code generation with test suites, or math tutoring with answer checking), this loop is implementable today using the paper's hyperparameters and training recipe.

Budget-constrained post-training for open-weight model releases. Teams releasing open-weight models often face a resource constraint: they want to release the best possible model but can only afford a limited post-training budget (either because training a large RL pipeline is expensive or because high-quality demonstration data is scarce). SD-ZERO offers a cost-effective post-training stage that requires only binary answer verification — no external teacher API calls (as would be needed for distillation from GPT-4 or Claude), no human annotation of reasoning traces, and no step-level correctness labels. The paper's total token budget of ≤225M tokens for both phases (Appendix C.2) translates to a concrete dollar amount at current cloud GPU prices: on 4-8 GPUs, the full two-phase pipeline completes within hours to a day depending on model size. For a team releasing a 7B-parameter math/code model, applying SD-ZERO as the final post-training stage — instead of or in addition to GRPO — could yield the +10% accuracy improvement reported in Table 1 at a compute cost comparable to running GRPO for one epoch on the same data. The SRT model alone (Phase 1) already achieves most of the gain (+7.8% for Qwen3-4B, +9.2% for Olmo-3-7B) and can serve as a strong baseline if the full two-phase pipeline is too expensive.

When to Prefer This Method

The paper explicitly positions SD-ZERO against RLVR (GRPO), supervised fine-tuning on external demonstrations (SFT), rejection fine-tuning (RFT), and self-distillation with privileged information (SDFT). The decision rule that emerges from the experimental results is:

• Prefer SD-ZERO over GRPO when training sample budget is constrained and the base model has non-trivial accuracy (≥10-20% on the target task). Under matched single-epoch generation budgets, SD-ZERO outperforms GRPO by 7.2 points on average for Qwen3-4B-Instruct (60.3% vs. 53.1%, Table 1), and doubling GRPO's sampling budget to 8 rollouts closes only 0.6 points of this gap (Table 10). The advantage is largest on harder benchmarks (AIME25: +10.0 points; HMMT25: +15.0 points; LiveCodeBench: +20.0 points), suggesting SD-ZERO is especially preferable when the base model struggles and needs richer per-sample supervision.

• Prefer SD-ZERO over SFT on external demonstrations when the external teacher's output distribution differs from the student model's, or when expert demonstrations are unavailable. The paper shows that SFT on DeepSeek-R1 traces degrades Qwen3-4B-Instruct on AMOBench (−2.5 points) and LiveCodeBench (−4.6 points, Table 1), while SD-ZERO improves on all benchmarks. SRT alone, trained on 6K self-generated traces, outperforms SFT on 15K external traces by 7.6 points on average, confirming that on-policy self-revision data is more effective than off-policy teacher data when distribution mismatch exists.

• Prefer SD-ZERO over RFT when the base model makes frequent errors that contain learnable patterns. RFT discards all incorrect responses, training only on correct ones; SRT preserves incorrect attempts as context for revision and achieves substantially larger gains on hard benchmarks (AIME25: +7.1 points over RFT; AMOBench: +4.7 points; LiveCodeBench: +6.4 points, Table 1). The harder the task (and thus the more incorrect responses the base model generates), the larger SRT's advantage over RFT, because RFT throws away exactly the data that SRT leverages.

• Prefer SD-ZERO over SDFT when you lack gold solution traces for the training data. The paper demonstrates this directly: SDFT given only final-answer supervision (no solution traces) performs at 49.5% on math benchmarks, essentially identical to the base model's 48.1% and far below SD-ZERO's 57.3% (Table 8). SDFT's mechanism depends on the teacher conditioning on a correct solution; SD-ZERO's teacher conditions on the student's (possibly incorrect) response and its binary reward, a capability explicitly trained in Phase 1.

• Prefer GRPO, SFT, or RFT over SD-ZERO when the task domain lacks reliable binary verification (the paper provides no evidence that SD-ZERO works without verifiable rewards), when deploying thinking models that generate extended chains of thought (the paper provides negative evidence in Table 14 and explicitly defers this to future work), or when multi-epoch training on large datasets is feasible and the asymptotic performance ceiling matters more than single-epoch sample efficiency (the paper's comparisons are single-epoch and do not establish SD-ZERO's ceiling relative to multi-epoch baselines).