SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
A user will ask you to solve a task. You should first draft your thinking
process (inner monologue). Then, generate the solution.
Your response format must follow the template below:
  thinking
Your thoughts or/and draft, like working through an exercise on scratch
paper. Be as casual and as long as you want until you are confident to
generate a correct solution.
  response
<solution>
Final solution presented to the user.
</solution><|eot_id|><|start_header_id|>user<|end_header_id|>
We are currently solving the following issue within our repository. Here is
the issue text:
--- BEGIN ISSUE ---
{problem_statement}
--- END ISSUE ---
Below are some code segments, each from a relevant file. One or more of
these files may contain bugs.
--- BEGIN FILE ---

--- END FILE ---
Please first localize the bug based on the issue statement, and then
generate *SEARCH/REPLACE* edits to fix the issue.
Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE
Here is an example:
```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE

Several design choices in this template are significant:

**The thinking/response separation.** The template enforces a structural separation between reasoning and code generation. The ``  thinking `` block is described as "as casual and as long as you want"—the model is encouraged to explore, reflect, and draft without space constraints, mimicking the scratch-paper behavior observed in DeepSeek-R1's "aha moments." The ``  response `` block contains only the final solution. This separation is not enforced programmatically during training (the model could, in principle, ignore it), but the reward function only extracts the patch from the ``<solution>`` tags within the ``  response `` block, so generating the patch outside this structure results in a format error reward of −1. This creates a soft incentive: the model is free to use the thinking block however it wants, but the patch must be correctly formatted.

**The search/replace format.** The model must output edits in a specific search/replace format adapted from Agentless (Xia et al., 2024). Each edit block specifies: the file path, a `<<<<<<< SEARCH` marker, the exact code lines to find in the existing file (with proper indentation preserved), a `=======` divider, the replacement lines, and a `>>>>>>>> REPLACE` marker. This format was chosen over alternative patch formats (unified diff, git diff, full file rewrites) because it is both human-readable and machine-parseable: the search block can be located in the original file via exact string matching, and the replace block can be substituted in. The template explicitly warns about proper indentation—this is because indentation errors in search blocks cause the search to fail to match the target code, making the entire edit inapplicable.

**Full file contents in context.** The template provides "each relevant file" in its entirety, identified by `--- BEGIN FILE ---` and `--- END FILE ---` markers. This is a deliberate departure from sparse retrieval-based approaches that only show a few relevant lines. By providing the complete file, the model is forced to identify the precise bug location within potentially hundreds of lines of code. This design choice is crucial for two reasons: (1) it teaches the model to perform fine-grained fault localization as an *internal reasoning step* within its thinking block, rather than relying on an external retriever, and (2) it ensures the context for understanding the bug (surrounding function signatures, imports, class definitions) is always available.

**Explicit "multiple SEARCH/REPLACE edits" instruction.** The template states that multiple edit blocks should be placed in separate code blocks. This teaches the model that real-world fixes often touch multiple locations and that the format supports an arbitrary number of edits per issue. This is important because SWE-bench issues often require changes in 2-5 different locations across one or more files.

---

#### Reward Modeling (Section 2.1, Equation 1)

The reward function is the central technical innovation of SWE-RL, and its design reflects the key insight that **continuous similarity to the oracle patch provides a better learning signal than discrete correctness in the messy, diverse world of real-world bug fixes**.

The reward function is defined formally as:

$$R(o) = \begin{cases} -1, & \text{if } o \text{ has wrong format} \\ \text{compare}(\text{patch}_{\text{pred}}, \text{patch}_{\text{gt}}), & \text{otherwise} \end{cases}$$

where `o` is the complete model rollout (the full generated text including both thinking and response), `patch_pred` is the patch extracted from the model's output if it follows the correct format (containing the search/replace edit blocks), and `patch_gt` is the oracle patch from the merged PR.

**What it computes:** The reward function first checks whether the model's output is correctly formatted—specifically, whether the response contains properly structured search/replace edit blocks with valid file paths, search strings, and replace strings. If the format is incorrect (e.g., the model failed to close a code block, omitted the file path, or wrote the patch outside the ``<solution>`` tags), the reward is exactly −1, a hard penalty that strongly discourages malformed outputs. If the format is correct, the predicted patch is extracted and compared to the oracle patch using Python's `difflib.SequenceMatcher`, which returns a floating-point value between 0 and 1. A score of 0 means the patches share no common subsequences—completely different changes. A score of 1 means the patches are identical at the sequence level. Intermediate scores capture partial matches: for example, if the model correctly identifies one of two required changes but misses the other, the similarity might be 0.4–0.6 depending on how much of the correct change is present.

**The `compare` function in detail.** `difflib.SequenceMatcher` is based on the Ratcliff-Obershelp algorithm (Ratcliff and Metzener, 1988). It computes the similarity of two sequences as:

$$\text{similarity} = \frac{2 \times M}{T}$$

where `M` is the number of matching characters (or tokens, depending on the sequence representation) and `T` is the total number of characters in both sequences. The factor of 2 normalizes the score to [0, 1]: when the sequences are identical, `M = T/2` (each character in the first sequence matches a character in the second), giving similarity 1; when the sequences share no common elements, `M = 0`, giving similarity 0. For SWE-RL, the sequences being compared are the string representations of the predicted and ground-truth patches—the search/replace blocks as text.

**Why this form (continuous similarity) over alternatives:**

**Alternative 1: Exact match (discrete, 0/1 reward).** The most obvious alternative is to reward 1 if the predicted patch exactly matches the oracle patch and 0 otherwise, analogously to how DeepSeek-R1 rewards exact answer matches in math. The paper explicitly tests this alternative in the reward ablation (Section 3.6, Figure 5). The discrete reward fails badly: training rewards remain near zero throughout, and the final repair performance is 29.0% (oracle files, greedy decoding) compared to 34.8% for the continuous reward. The failure mode is instructive: real-world patches have enormous diversity. Two developers fixing the same bug might produce syntactically different but semantically equivalent patches—one might add an early return, another might restructure a conditional. Even when the fix is identical, variable names, whitespace, and comment styles can differ. An exact-match reward treats all of these as equally wrong (reward 0), providing no gradient to guide the model toward progressively better solutions. The model cannot learn which of two incorrect patches is "closer" to correct, so optimization stalls.

**Alternative 2: Execution feedback.** In competitive programming, the standard approach is to execute the generated code against test cases and reward passing tests (DeepSeek-AI, 2025; Gehring et al., 2025). For SWE-bench, this is infeasible for several reasons: (1) setting up execution environments for thousands of diverse Python repositories with different dependency chains is computationally prohibitive, (2) even when an environment can be set up, reproduction tests that reliably trigger the bug and verify the fix are not guaranteed to exist (the issue reporter may not have provided a minimal reproduction), and (3) passing existing tests is necessary but not sufficient—a patch that silences the error by deleting the problematic code might pass all tests but is clearly incorrect. The similarity reward sidesteps these problems entirely by operating purely on the textual representation of the patch, requiring no execution whatsoever.

**Alternative 3: Semantic equivalence (AST-based comparison).** One could compare patches by analyzing their effect on the abstract syntax tree—do they make the same changes to the same AST nodes? This would handle syntactic variations better than string matching. However, AST-based comparison requires language-specific parsers, fails on syntactically invalid intermediate patches (common during RL exploration), and cannot capture changes to non-code files (configuration, documentation) that are sometimes part of real fixes. The sequence matcher is language-agnostic, works on any text, and provides a smooth, continuous signal even for partially correct patches.

**The −1 format penalty.** The choice of −1 rather than 0 for format errors is deliberate. A 0 reward for format errors would place malformed patches on the same footing as perfectly formatted but completely irrelevant patches, providing no disincentive against format violations. A −1 penalty ensures that format errors are strictly worse than any correctly formatted patch (which gets at minimum 0 similarity), creating a strong gradient toward proper formatting. The paper reports format accuracy of 95.6% for the continuous reward model (Table 2), compared to 12.2% for the base Llama-3.3-70B-Instruct, confirming that the penalty effectively teaches format compliance.

**The reward operates on complete patches, not per-token.** Unlike process reward models that score intermediate steps (as in the example paper's search methods), SWE-RL's reward is outcome-based: the model receives one scalar reward for the entire generated patch. This is analogous to DeepSeek-R1's approach of rewarding only the final answer. The model must learn, through GRPO's credit assignment across the full rollout, which aspects of its thinking and generation process contributed to patch quality. This simplicity is a feature—no step-level annotation is required, and the reward can be computed automatically for any PR with a merged patch.

---

#### Group Relative Policy Optimization (Section 2.1, Equation 2)

The optimization algorithm that converts the similarity rewards into policy updates is Group Relative Policy Optimization (GRPO) (Shao et al., 2024), chosen over standard PPO for its stability and reduced computational requirements. The GRPO objective is:

$$J(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^G \left(\min\left(\frac{\pi_{\theta}(o_i \mid q)}{\pi_{\theta_{\text{old}}}(o_i \mid q)} A_i, \text{clip}\left(\frac{\pi_{\theta}(o_i \mid q)}{\pi_{\theta_{\text{old}}}(o_i \mid q)}, 1 - \epsilon, 1 + \epsilon\right) A_i\right) - \beta D_{\text{KL}}(\pi_{\theta} \parallel \pi_{\text{ref}})\right)\right]$$

where the expectation is taken over `(issue, ctx, patch_gt) ~ D_seed`, the prompt is `q = form_prompt(issue, ctx)`, and `{o_i}_{i=1}^G ~ π_{θ_old}(· | q)` are `G = 16` rollouts sampled from the old policy.

**Symbol definitions:**
- `D_seed` is the 273,000-instance seed RL dataset, where each instance is a triple of issue description, code context, and oracle patch.
- `q` is the formatted prompt derived from the issue and context via the template in Figure 2.
- `π_θ` is the current policy (the model being optimized), parameterized by θ.
- `π_{θ_old}` is the policy from the previous optimization step, frozen during the current step's advantage computation.
- `π_ref` is a reference policy—the initial Llama-3.3-70B-Instruct checkpoint, frozen throughout training.
- `o_i` is the i-th rollout (full generated text) for prompt `q`, sampled from the old policy.
- `G = 16` is the group size—the number of candidate solutions generated per prompt. The paper uses 32 prompts per global batch (global batch size 512 / 16 rollouts per prompt = 32), so the 512 rollouts represent 32 problems × 16 candidates each.
- `A_i = (r_i - mean(r_1, ..., r_G)) / std(r_1, ..., r_G)` is the advantage for rollout `i`, computed as the normalized reward within its group of G candidates. The normalization subtracts the group mean and divides by the group standard deviation.
- `r_i = R(o_i)` is the reward for rollout `i`, computed by the reward function (Equation 1).
- `ε` is the clipping parameter (standard PPO value, typically 0.2, though the paper does not specify the exact value used).
- `β` is the KL penalty coefficient, controlling how strongly the policy is penalized for deviating from the reference model.
- `D_KL(π_θ ∥ π_ref)` is the estimated Kullback-Leibler divergence between the current policy and the reference policy, approximated using the method from Schulman (2020).

**What it computes:** For each batch, the algorithm: (1) samples 16 candidate solutions for each of 32 problems from the old policy; (2) computes the reward for each candidate using the similarity reward function; (3) normalizes rewards within each group of 16 into advantages (so a reward of 0.5 might become advantage +1.2 if the group mean is 0.2 and standard deviation is 0.25, or advantage −0.8 if the group mean is 0.7 and standard deviation is 0.25); (4) computes the probability ratio `π_θ(o_i | q) / π_{θ_old}(o_i | q)`—how much more (or less) likely the current policy is to generate this rollout compared to the old policy; (5) multiplies the ratio by the advantage and clips the ratio to [1−ε, 1+ε] to prevent destructively large updates (the standard PPO clipped surrogate objective); (6) takes the minimum of the clipped and unclipped objectives (pessimistic bound, standard in PPO); (7) subtracts `β × D_KL` to penalize the policy for moving too far from the reference model; (8) averages over the G candidates and over all prompts in the batch; (9) performs a single Adam optimization step to maximize this objective.

**Why This Form (GRPO Over Standard PPO):**

**Group-based advantage normalization replaces the learned value function.** Standard PPO requires a separate value network (critic) to estimate the expected return for each state, which is used to compute advantages `A = r - V(s)`. This critic must be trained alongside the policy and must be roughly the same size as the policy model for large LLMs—doubling the memory and compute requirements. GRPO eliminates the critic entirely by using the group mean as an empirical baseline: if a rollout's reward is above the average of its peer rollouts for the same prompt, its advantage is positive; if below average, its advantage is negative. This works because rollouts within a group are independent samples from the same policy for the same prompt, so their rewards form an unbiased (though noisy) estimate of the expected reward for that prompt. The normalization by standard deviation makes the advantages unitless and roughly scale-invariant, preventing reward magnitude differences across prompts from causing uneven policy updates.

**KL penalty against the reference model prevents reward hacking and language degradation.** A well-known failure mode in RL fine-tuning of LLMs is that the policy learns to exploit the reward function—producing outputs that score highly under the reward but are nonsensical, repetitive, or ungrammatical. The KL penalty `β D_KL(π_θ ∥ π_ref)` directly penalizes the policy for generating token distributions that diverge from the reference model (the original Llama-3.3-70B-Instruct). This serves two purposes: (1) it keeps the model's language capabilities intact, preventing the "language drift" where the model forgets how to produce fluent English text, and (2) it prevents the model from collapsing to a degenerate strategy of always outputting the same high-reward pattern regardless of the input. The reference model acts as an anchor, and β controls how far the policy is allowed to sail from that anchor.

**Clipping prevents destructively large policy updates.** The `clip(ratio, 1−ε, 1+ε)` term ensures that no single rollout can change the policy's probability of generating that rollout by more than a factor of `1+ε` in a single step (for positive advantages) or `1−ε` (for negative advantages). Without clipping, a rollout that the old policy assigned near-zero probability but the new policy assigns high probability (ratio ≫ 1) would produce an enormous gradient that could destabilize training. The `min(ratio × A, clipped_ratio × A)` construction is the standard PPO pessimistic bound: for positive advantages, the objective is capped at `(1+ε) × A`, preventing the policy from increasing probability too aggressively; for negative advantages, the objective is floored at `(1−ε) × A`, preventing the policy from decreasing probability too aggressively.

**Training configuration details (Section 3.1).** The paper reports specific hyperparameters:
- **Training steps:** 1,600 global steps, each consisting of one optimization update.
- **Context window:** 16,000 tokens (16k), which accommodates the full contents of multiple code files plus the issue description and the model's thinking block.
- **Global batch size:** 512 rollouts per step.
- **Rollouts per problem:** G = 16, meaning 32 distinct problems per global batch (32 × 16 = 512).
- **Optimizer:** Adam (Kingma and Ba, 2017). The paper does not specify the learning rate, β₁, β₂, or weight decay values in the main text.
- **Hardware:** Training uses 512 NVIDIA H100 GPUs, taking approximately 32 wall-time hours per training run.
- **Policy initialization:** The policy `π_θ` is initialized from Llama-3.3-70B-Instruct, and the reference model `π_ref` is the same frozen checkpoint.

**The sampling and update cycle.** At each global step, the current policy (old policy for that step) generates 16 candidate solutions for each of 32 randomly sampled prompts from `D_seed`. The rewards are computed using Equation 1. The advantages are computed using group normalization. Then a single Adam step updates the policy parameters to maximize the GRPO objective. The updated policy becomes the old policy for the next step. This means the policy's own generations are always on-policy relative to the most recent update, providing a tight coupling between the data distribution and the optimization—the model is always training on its own mistakes at the current capability level.

---

#### The Agentless Mini Scaffold (Appendix B, Figure 7)

While SWE-RL training involves only a single task—given issue description and full file contents, produce search/replace edits—the SWE-bench evaluation requires additional capabilities that the model is never explicitly trained on: file-level localization (identifying which files in a repository are relevant), reproduction test generation (writing tests that reproduce the bug and verify the fix), and regression test selection (identifying existing tests that should continue passing). The Agentless Mini scaffold (Figure 7) wraps the RL-trained model in a multi-step pipeline that decomposes SWE-bench evaluation into these subtasks, relying on the model's emergent generalization to handle steps it wasn't trained for.

The scaffold has four sequential stages, each producing outputs that feed into the next:

**Stage 1: Localization and repair.** This is the only stage that directly leverages the RL-trained model's core capability. The localization step uses a simplified prompting approach compared to the original Agentless scaffold: the model receives the issue description and the repository's directory structure (not file contents) and generates multiple candidate lists of potentially relevant file paths. Unlike Agentless, which involves two intermediate steps (identifying related elements and then related files) plus a separate embedding-based retrieval model, Agentless Mini collapses localization into a single generation step. The model produces multiple samples (e.g., 10–50 candidate file sets), which are then deduplicated into unique sets. Each unique file set becomes the input to the repair step: the model receives the issue description plus the *full contents* of all files in the set and generates search/replace edits. Multiple repair samples are generated from different location sets (e.g., if there are 5 unique file sets and 100 repair samples per set, that yields 500 patches total), ensuring comprehensive exploration of the patch search space. The key insight is that the RL-trained model's ability to perform fine-grained fault localization within full file contents—developed during training when it had to identify bug locations in large code context—transfers to the coarser file-level localization task even though it was never trained on directory structure inputs.

**Stage 2: Reproduction tests generation and selection.** Multiple reproduction test samples are generated based on the issue description. Each test must implement specific logic: it should detect whether the issue is currently present (outputting "Issue reproduced") and whether it has been fixed (outputting "Issue resolved"). Two enhancements over the original Agentless scaffold are implemented: (1) the model is first prompted to predict a relevant test file path (using the repository structure) to guide test generation toward the appropriate test module, and (2) rather than selecting a single majority-voted test, the scaffold selects multiple top-ranked test samples based on voting results. The selected tests are filtered by executing them against the unmodified codebase—only tests that correctly output "Issue reproduced" (confirming they can detect the bug) are retained. The paper's scaling analysis (Figure 4, right) shows that increasing the number of reproduction tests from 1 to 20 steadily improves final pass@1, with diminishing returns after 20 tests.

**Stage 3: Regression tests selection.** This stage mirrors the original Agentless approach and requires no model inference in its minimal form. All tests in the repository are executed before applying any patches, and those that pass are collected. An optional inference step can filter out passing tests that are *expected* to fail after the bug is fixed (e.g., tests that were added specifically to capture the known bug)—these are excluded from the regression set. The remaining passing tests become the regression test suite: patches that cause any of these tests to fail are penalized during reranking.

**Stage 4: Reranking.** The final stage selects the best patch from the potentially hundreds of candidates generated in Stage 1. The process works as follows:

1. **Regression test execution:** Every candidate patch is applied to the repository and the regression test suite from Stage 3 is executed. Patches that cause the fewest regression test failures are preferred. This filters out patches that fix the target bug but introduce regressions.
2. **Reproduction test execution:** For each patch, the top-N reproduction tests from Stage 2 are executed. A test is marked as "passing" for a patch if it outputs "Issue resolved" when run against the patched codebase.
3. **Dual execution agreement:** The CodeT (Chen et al., 2023) dual execution agreement objective is applied. Let `P` be the set of patches that pass exactly the same set `T` of reproduction tests—these form a consensus group. The group is scored as `|P| × |T|²`. This scoring function has two important properties: (a) groups where patches pass *more* tests score exponentially higher (the `|T|²` factor prioritizes test coverage over consensus size), and (b) among groups passing the same tests, groups with *more* agreeing patches score linearly higher (the `|P|` factor breaks ties in favor of larger consensus). This incentivizes the selection of patches that both pass many tests and have convergent support from multiple independently generated candidates.
4. **Final selection:** The consensus group with the highest score is selected, and within that group, the best patch is chosen by majority voting (the patch that was generated most frequently within the group).

**Why this scaffold design matters for understanding SWE-RL's contribution.** The scaffold decouples the model's learned capability from the evaluation mechanics. The RL training teaches the model a single competency: given issue + full code → produce a correct patch with reasoning. The scaffold handles everything else—file retrieval, test generation, execution-based verification, consensus-based selection—without relying on the RL model for these steps. This means the reported 41.0% solve rate is a lower bound on the RL model's capability: it measures how well the model's core repair competency can be leveraged by an external pipeline, not how well the model can autonomously navigate a repository. The paper's claim that the RL model generalizes to untrained subtasks (file-level localization, test generation) is evidenced by the fact that the scaffold *works at all*—the model must produce reasonable outputs for these untrained steps, even though the scaffold does not use execution feedback or RL rewards to optimize those steps specifically.

---

#### SFT Baseline Construction (Appendix C, Figure 8)

To isolate the effect of RL from the effect of simply training on more software engineering data, the paper constructs a carefully controlled supervised fine-tuning baseline, Llama3-SWE-SFT-70B. The SFT pipeline mirrors the RL pipeline in its data sources but differs in how it generates training examples and what objective it optimizes.

**Data sources differ fundamentally.** While SWE-RL uses the raw PR data directly as an environment for exploration and reward, SFT requires demonstration data—examples of "what a good output looks like" for each input. The SFT data is generated using a Magicoder-style (Wei et al., 2024) synthetic data pipeline:

1. **Seed collection:** High-quality PR seeds are selected from the same 273,000-instance pool using similar heuristics (linked issue, bug-fix nature, programming file changes).
2. **Synthetic localization data:** Llama-3.3-70B-Instruct is prompted with the issue description, repository structure, and paths of edited and relevant files. The model generates a thought process followed by a prioritized list of file paths. These synthetic outputs are filtered: the model's response must include all files that were actually edited in the PR, and these files must appear toward the top of the ranking.
3. **Synthetic code editing data:** Llama-3.3-70B-Instruct is prompted with the issue, the ground-truth PR, and the oracle patch as guidance, and generates code edits in search/replace format. The outputs are filtered to ensure correct formatting and that all search blocks match actual code in the input files.

**Training data mixture.** Unlike the RL model, which trains on only the seed PR dataset, the SFT model is trained on a mixture of: (a) the synthetic localization and editing data described above, (b) coding SFT data from the Llama 3 training distribution, and (c) general SFT data from the Llama 3 training distribution. This mixture is necessary because SFT on issue-fixing data alone would cause severe overfitting and loss of general capabilities. The SFT model is trained for 2 billion tokens with a 16k context window.

**Why SFT leads to overfitting while RL leads to generalization (the paper's interpretation).** The paper argues that the SFT objective steers the model to mimic the specific output distribution of the training data—format, style, reasoning patterns—rather than developing generalizable problem-solving strategies. Even with a carefully curated mixture of general data, SFT on a domain-specific task tends to suppress capabilities on tasks that receive less training emphasis. RL, by contrast, optimizes for an outcome (patch similarity) while allowing the model to discover its own internal strategies for achieving that outcome. The KL penalty against the reference model further prevents the RL model from drifting too far from its original general capabilities. This is the core insight behind the paper's generalization results (Table 3): SFT teaches the model *what outputs to produce*, while RL teaches the model *how to reason to solve problems*, and the latter transfers across domains while the former does not.

**Key difference in computational cost.** The RL training is simpler in one important way: it requires only the seed PR dataset (issue, context, oracle patch), with no synthetic data generation step. The SFT pipeline requires running a separate generation pass using Llama-3.3-70B-Instruct to create the localization and editing demonstrations, plus filtering these outputs against ground truth. This makes the RL approach both more autonomous (no teacher model needed) and potentially more scalable (the data pipeline is simpler, and the model generates its own training signal through exploration).

---

#### Summary of Design Choices and Their Justifications

- **Continuous similarity reward over exact match:** Real-world patches are too diverse for exact matching to provide a useful learning signal. `difflib.SequenceMatcher` provides a smooth gradient from "completely wrong" to "partially correct" to "fully correct," enabling the model to learn from partially correct attempts. The reward ablation (Figure 5) confirms that continuous rewards produce substantially better final performance (34.8% vs. 29.0% repair accuracy) and faster reward growth during training.

- **Merge-base file retrieval over main-branch snapshots:** The code context must reflect what the developer actually saw when writing the fix, not the later state of the main branch. Using the merge base prevents training on stale or inconsistent code states that would confuse the model about what needs changing.

- **Full file contents in context over snippet-based retrieval:** Providing complete files forces the model to develop internal fault localization reasoning (identifying which specific lines are buggy) rather than relying on an external retriever. This skill transfers to out-of-domain tasks requiring detailed code understanding.

- **Search/replace edit format over unified diff or full rewrites:** The search/replace format is both human-readable and machine-parseable, with explicit markers that make format validation straightforward (enabling the −1 penalty for format errors) while allowing the model to express arbitrary edits.

- **GRPO over standard PPO:** Group-based advantage normalization eliminates the need for a separate value network (critic), halving the memory requirements for training a 70B model on 512 H100 GPUs. The KL penalty against the reference model prevents language drift and reward hacking.

- **Relevant-but-unchanged files in context (with no edits required):** Without these negative examples, the model learns a bias that every file in the context needs editing. Including distractor files that must not be edited teaches the model to distinguish "files to read for understanding" from "files to modify."

- **Pipeline-based scaffold (Agentless Mini) over agentic scaffold:** A pipeline with a single repair step simplifies the RL training to a single task (issue + code → patch), while the scaffold handles the multi-step evaluation pipeline. The paper shows that RL on this single task generalizes to the scaffold's other steps (file localization, test generation) without explicit training.

- **Predicted difficulty via PRM score distribution (vs. oracle pass@1):** Not applicable—SWE-RL does not use difficulty estimation. All training instances are treated uniformly, and the GRPO advantage normalization implicitly handles varying difficulty by computing advantages relative to other attempts on the same problem.

## 4. Key Insights and Innovations

### Innovation 1: Continuous Sequence Similarity as a Sufficient Reward Signal for Real-World Patch Generation

The field's dominant assumption—established by DeepSeek-R1 and its follow-ups—has been that reinforcement learning for coding and reasoning requires *automatically verifiable correctness*: execution feedback from test cases for code, or exact string matching for math answers. SWE-RL challenges this assumption at a fundamental level by demonstrating that a much weaker signal—the `difflib.SequenceMatcher` similarity between a predicted patch and the ground-truth developer patch—is not only sufficient but *more effective* than exact-match rewards for learning real-world bug fixing.

This is not an incremental improvement to existing reward designs; it is a **conceptual reframing** of what constitutes a viable training signal for complex software engineering tasks. The logic is counterintuitive: a continuous similarity reward between 0 and 1 provides a richer learning gradient than a discrete 0/1 correctness signal, precisely *because* real patches are too diverse for exact matching. When two developers can fix the same bug with syntactically different but semantically equivalent patches, an exact-match reward treats all correct-but-different solutions as failures (reward 0), providing no gradient to distinguish "almost right" from "completely wrong." The sequence matcher, by crediting partial overlap—correct file paths, correct function signatures, correct edit locations even if the edit content differs—gives the model a path to climb from random guesses to targeted fixes.

The paper's reward ablation (Section 3.6, Figure 5) is the key evidence: the discrete reward variant plateaus at near-zero average reward and achieves only 29.0% repair accuracy, while the continuous reward variant shows steadily rising rewards throughout training and reaches 34.8%. This gap—nearly 6 percentage points on a task where the base model scores 5.4%—is the difference between "RL works for software engineering" and "RL fails for software engineering." The discrete variant's training dynamics (rewards remaining near zero through 1,600 steps) reveal that the model never receives enough positive signal to escape random exploration, while the continuous variant's rising curve shows the model accumulating partial credit and progressively refining its outputs.

What makes this insight **fundamental rather than incremental** is that it opens RL-based training for an entire class of tasks where execution-based verification is infeasible: any domain where the ground truth is a structured artifact with meaningful intermediate similarity to incorrect attempts—document editing, configuration management, refactoring, bug report generation—can potentially adopt this reward paradigm. The paper demonstrates that you don't need to solve the hard problem of semantic equivalence; sequence similarity is good enough, and its very "weakness" (crediting partial matches that aren't semantically correct yet) is what makes it a strong learning signal by smoothing the reward landscape.

This also explains a puzzle in the DeepSeek-R1 technical report, which noted limited effectiveness on SE tasks: they likely used exact-match or execution-based rewards, which fail for exactly the reasons this ablation demonstrates. SWE-RL's similarity reward is the missing ingredient that makes RL viable for real-world software engineering.

---

### Innovation 2: RL on Software Evolution Data Produces Generalized Reasoning While SFT on the Same Seeds Produces Task-Specific Overfitting

The paper's most surprising and intellectually significant finding is not the SWE-bench performance (41.0% is strong but might be expected from a well-tuned training pipeline) but the **diametrically opposite generalization behavior** of RL-trained and SFT-trained models when evaluated on out-of-domain tasks. The SFT baseline degrades relative to the base Llama-3.3-70B-Instruct on four out of five OOD benchmarks (Table 3: drops of 3.0 points on HumanEval+, 4.8 on BigCodeBench-Hard (C), 6.1 on MATH strict, and 1.2 on MMLU), while the RL model *improves* on all five (gains of 3.7, 0.0, 0.0, 10.5, and 0.3 respectively). This is a clean dissociation: same base model, trained on data derived from the same seed PRs, but RL yields positive transfer and SFT yields negative transfer.

This finding constitutes a **diagnostic contribution** about the nature of different training objectives, not merely a performance comparison. It suggests that SFT and RL are not two interchangeable ways to incorporate domain-specific data—they are fundamentally different learning paradigms with different inductive biases about what the model should retain from its pretraining. SFT teaches the model to match a specific output distribution, and even with a "meticulously curated data mix" of general coding and dialog data (as the paper notes), the optimization pressure toward the target distribution dominates, causing the model to partially unlearn capabilities that are underrepresented in the SFT mixture. RL, constrained by the KL penalty against the frozen reference model, teaches the model *how to reason toward solutions* within a bounded deviation from its original behavior, preserving general capabilities while enhancing the reasoning strategies that produce correct outputs.

The significance of this finding extends beyond software engineering. If the pattern generalizes—RL on domain-specific outcome rewards preserves or enhances general capabilities while SFT on domain-specific demonstrations erodes them—then RL is the **strictly preferable paradigm** for incorporating new capabilities into pretrained LLMs whenever a reward function can be defined. It implies that the current dominant approach in the field (distillation from stronger models via SFT) is not just limited by teacher quality (as the paper notes in Section 2) but actively harmful to the model's broader competence. The paper doesn't claim this generalizes to all domains, but the evidence in Table 3—spanning code generation, code reasoning, mathematics, and language understanding—makes a compelling case that the effect is not task-specific.

This is simultaneously a **negative result with positive implications**: SFT degrades OOD performance (negative), but the degradation is not inevitable—RL avoids it (positive). This reframes the choice of training methodology from an implementation detail to a first-order design decision with consequences for the model's entire capability profile.

The qualitative evidence in Figure 3 supports this interpretation: the RL model exhibits "aha moments" with self-reflection, exploration of multiple approaches, and divide-and-conquer strategies even on out-of-domain tasks like function implementation and mathematics. These reasoning patterns emerge from optimizing patch similarity—the model discovers that spending more tokens on structured deliberation produces higher rewards—and the patterns transfer because they are *process-level strategies* rather than *content-level patterns*. SFT, by contrast, teaches the model to reproduce the surface form of chain-of-thought without necessarily internalizing the underlying reasoning logic, leading to brittleness when the task format or domain shifts.

---

### Innovation 3: The Merge-Base Retrieval and Relevant-File Distractor Strategy as Implicit Curriculum Design for Fault Localization

While the data curation details in Appendix A might appear to be engineering, they embody a **non-obvious instructional design choice** that fundamentally shapes what the model learns. The decision to retrieve file contents at the merge base of the PR (not the main branch head) and to include relevant-but-unchanged files as distractors is not merely about data quality—it creates an implicit curriculum that teaches fault localization without any explicit localization supervision.

Prior work on training models for repository-level tasks has generally followed one of two patterns: (1) use retrieval-based localization as a preprocessing step, reducing the model's task to "given the correct file, produce the fix" (as in the original Agentless scaffold's multi-step localization), or (2) train separate localization and repair models, treating them as distinct capabilities. SWE-RL's approach is different: by presenting the model with *full file contents for multiple files*—only some of which need editing—and requiring it to produce edits only for the correct files, the RL process forces the model to learn localization as an internal reasoning step. The model cannot succeed by editing everything it sees (the distractor files teach it not to), and it cannot succeed by random guessing (the continuous reward penalizes edits to wrong files by reducing similarity to the oracle patch, which only contains changes to the correct files).

This is a **curriculum design insight** rather than an algorithmic one: what you include in the context, and what you hold out for reward computation, jointly define the learning problem. The merge-base decision ensures the model sees code states that *actually contained the bug*, which is essential for learning to recognize bug indicators. The distractor file decision ensures the model learns to distinguish "files to read" from "files to edit"—a capability that the original SWE-bench paper's fine-tuning experiments found was not learned when all context files required edits.

The significance is that this approach eliminates the need for a separate localization training phase or a localization-specific reward component. The model's emergent ability to perform file-level localization in the Agentless Mini scaffold (Stage 1) is a direct consequence of the training design, even though the model was never shown a repository directory structure or asked to rank files during RL. The paper's evidence for this is implicit but clear: the end-to-end pipeline works at 41.0% solve rate, which requires accurate localization in the first stage. If the model had not learned to localize from full-file contexts, it would fail at the localization step and never reach the repair step where its RL-trained capabilities are strongest.

This innovation is **incremental in mechanism** (data filtering and context construction are standard techniques) but **fundamental in implication**: it suggests that carefully designed RL environments can teach composite skills (localization + repair) through a single reward signal on the final output, without decomposing the task into separately trained subtasks. This is a form of *emergence through environmental design*—the model discovers localization as a necessary subgoal for maximizing patch similarity—that parallels how DeepSeek-R1 models discovered verification and self-correction behaviors from outcome-only math rewards.

---

### Innovation 4: The Decoupling of Training Simplicity from Evaluation Complexity via Scaffold Complementarity

SWE-RL introduces a **methodological principle** that is likely underappreciated in the current agent-focused software engineering literature: the model's training task should be as simple as possible (single-turn generation with full context), while the evaluation scaffold handles the complexity of multi-step repository interaction. This is not laziness—it is a deliberate strategy that the paper's results validate.

The dominant trend in SWE-bench research has been toward increasingly sophisticated agentic scaffolds (SWE-agent, OpenHands, AutoCodeRover) where the model makes sequential decisions with tool interactions, and training these agents requires either reinforcement learning in interactive environments (SWE-Gym) or distillation of agent trajectories from stronger models. This creates a tight coupling between training complexity and evaluation complexity: if you want the model to navigate a repository autonomously, you must train it to navigate a repository autonomously, which requires interactive RL environments or expensive trajectory data.

SWE-RL breaks this coupling. The training task is a single prompt → single completion: read the issue and the full code context, think, produce a patch. There are no tool calls, no multi-turn interactions, no environment to manage. The evaluation scaffold (Agentless Mini) then wraps this single-turn capability in a multi-step pipeline that handles repository navigation, test generation, and execution-based reranking. The model never learns to interact with a filesystem or execute code—it only learns to reason about code from static context—yet the composite system achieves state-of-the-art performance.

This principle has two significant implications. First, it **dramatically reduces the barrier to entry** for training models on real-world SE tasks. The SWE-RL training pipeline requires only a dataset of (issue, context, oracle patch) triples and a similarity function—no execution environments, no interactive simulators, no trajectory collection. Any research group with access to GitHub data and sufficient GPUs can replicate the approach. Second, it **separates capability improvement from scaffold innovation**: improvements to the model (through better RL techniques, larger models, more data) and improvements to the scaffold (better localization, smarter test generation, more sophisticated reranking) can proceed independently and compound. The paper's scaling analysis (Figure 4) demonstrates this directly: increasing repair samples (model capability) and reproduction tests (scaffold capability) both independently improve the final solve rate.

This is a **fundamental methodological contribution** disguised as an engineering choice. It challenges the implicit assumption in the agentic-SWE literature that training and evaluation must occur in the same modality, and it provides a template for future work: isolate the core reasoning task, train on it with simple rewards, and let an external scaffold handle the rest. The fact that the model generalizes to untrained scaffold steps (localization, test generation) is bonus evidence that this decoupling doesn't sacrifice capability—it may even enhance it by allowing the model to focus its learning capacity on the highest-value reasoning skill.

## 5. Experimental Analysis

### Evaluation Methodology

- **Dataset.** The primary evaluation is on SWE-bench Verified (OpenAI, 2024), a 500-instance subset of SWE-bench (Jimenez et al., 2023) where each instance has been human-verified to ensure the issue description is clear, the oracle patch correctly resolves the issue, and the evaluation tests are reliable. SWE-bench tests models on real-world GitHub issues from 12 popular Python repositories; the Verified subset removes instances where the original benchmark had ambiguous specifications or unreliable tests. The paper explicitly excludes all repositories used by SWE-bench from the training data curation pipeline to prevent contamination (Appendix A). For out-of-domain generalization evaluation (Section 3.5, Table 3), the paper uses HumanEval+ (Chen et al., 2021; Liu et al., 2023) for function-level code generation, BigCodeBench-Hard (Zhuo et al., 2024) for practical code generation with library use (reporting both Instruct and Completion variants), CRUXEval (Gu et al., 2024) for code execution reasoning (both Input and Output prediction), MATH (Hendrycks et al., 2021c) for mathematical reasoning, and MMLU (Hendrycks et al., 2021b) for general language understanding—all evaluated with zero-shot greedy decoding.

- **Base model(s).** The policy model is initialized from Llama-3.3-70B-Instruct (Dubey et al., 2024), a 70-billion parameter dense transformer. The paper chooses this model because it represents a strong open-source baseline with "representative" capabilities for contemporary LLMs, and its 70B scale makes RL training computationally feasible (512 H100 GPUs for 32 hours) while being small enough (<100B) to compare against other medium-sized models in Table 1. The reference model π_ref is the frozen initial Llama-3.3-70B-Instruct checkpoint, used for the KL penalty in GRPO. The paper also uses Llama-3.1-70B-Instruct as a separate model for generating relevant-but-unchanged file predictions during data curation (Appendix A).

- **Metrics.** The primary metric is pass@1 on SWE-bench Verified (the fraction of the 500 issues for which the model's selected patch passes all evaluation tests). For repair-only capability evaluation (Table 2), the metric is the repair pass rate with oracle files provided (i.e., how often the model's single greedy-decoded patch is correct when given the exact files that need editing). In the SFT baseline comparison (Table 2), format accuracy is reported as the percentage of model outputs that contain correctly structured search/replace edit blocks. For the OOD generalization experiments (Table 3), metrics follow each benchmark's standard convention: pass@1 for HumanEval+, BigCodeBench-Hard, and CRUXEval; macro-averaged accuracy across categories for MMLU; and strict (format-matching) and lenient (allowing `\boxed{}`) accuracy for MATH. For the reward ablation (Section 3.6, Figure 5), average training reward is reported as a continuous value tracking the model's progress, and repair (oracle) is the repair-only pass rate under greedy decoding.

- **Baselines.** The paper employs several categories of baselines:
  - **Base model without fine-tuning:** Llama-3.3-70B-Instruct evaluated with Agentless Mini (Table 1) and in repair-only mode with both greedy decoding and 20-sample majority voting (Table 2). The majority voting variant is specifically included because the base model's format accuracy is only 12.2% under greedy decoding, making it an unfair point of comparison without a format-correction mechanism.
  - **SFT baseline:** Llama3-SWE-SFT-70B, trained on the same Llama-3.3-70B-Instruct using a Mixture of synthetic code editing data generated in the Magicoder style, Llama 3 coding SFT data, and Llama 3 general SFT data (Section 3.1, Appendix C). This baseline is critical: it uses the same seed PR data as SWE-RL but processes it through SFT rather than RL, enabling a direct comparison of training paradigms. It achieves 36.2% on SWE-bench Verified with Agentless Mini (Table 1).
  - **Prior open-source methods (Table 1):** SWE-Llama-13B and SWE-Llama-7B (Jimenez et al., 2023) using RAG; Lingma-SWE-GPT-7B and Lingma-SWE-GPT-72B (Ma et al., 2024) using SWE-SynInfer; SWE-Gym-32B (Pan et al., 2024) using OpenHands; SWE-Fixer-72B (Xie et al., 2025) using SWE-Fixer. All of these are noted to incorporate distilled outputs from GPT-4o or Claude-3.5-Sonnet in their training data, making them teacher-dependent baselines.
  - **Proprietary model baselines (Table 1):** GPT-4o with SWE-agent (23.2%) and Agentless (38.8%), Claude-3.5-Sonnet with SWE-agent (33.6%), AutoCodeRover-v2.0 (46.2%), Tools (49.0%), OpenHands (53.0%), and Agentless (50.8%), o1-preview with Agentless (41.3%), DeepSeek-V3 with Agentless (42.0%), and DeepSeek-R1 with Agentless (49.2%). These establish the performance ceiling from proprietary and very large open models.
  - **Discrete reward ablation (Section 3.6, Figure 5):** A variant of Llama3-SWE-RL trained with a 0/1 exact-match reward instead of continuous similarity, matching the DeepSeek-R1 approach. This baseline tests whether the continuous similarity reward is necessary or merely incidental.

- **Generation budget / compute accounting.** For the RL training, compute is measured in global steps (1,600 total) with a batch size of 512 rollouts per step (32 problems × 16 candidates each), on 512 H100 GPUs for 32 hours. For SWE-bench evaluation, generation budget is measured by the number of repair samples per issue (swept from 40 to 500 in the scaling analysis, Figure 4 left) and the number of reproduction test samples (swept from 1 to 30, Figure 4 right). The main evaluation uses 500 repair samples and 30 reproduction tests at temperature 1.0. For the repair-only baseline evaluation (Table 2), a single greedy-decoded generation is used, except for the base Llama model which additionally uses 20-sample majority voting at temperature 0.6 to improve format accuracy. For OOD generalization evaluation (Table 3), all experiments use zero-shot greedy decoding. No FLOPs-matched pretraining comparison is performed—unlike the reference example paper, this work does not compare inference compute efficiency against pretraining compute scaling. The paper does not report inference FLOPs or latency for any evaluation.

- **Cross-validation / statistical protocol.** The paper acknowledges that it does not include error bars or confidence intervals for SWE-bench evaluations, citing the high per-instance cost and the benchmark's convention of single-attempt evaluation (NeurIPS checklist item 7). For the OOD generalization results (Table 3), the paper provides a statistical significance analysis: it references Eval Arena (Wang et al., 2024a) thresholds, noting that improvements of >0.8 percentage points on MMLU, >3 points on CRUXEval, and >3 points on full MATH are individually significant, and that taken together the aggregate evidence achieves significance at the 0.05 level. The paper does not describe any cross-validation procedure for hyperparameter selection or strategy selection, unlike the reference example paper's two-fold cross-validation within difficulty bins.

### Main Quantitative Results

#### Main Results on SWE-bench Verified (Table 1)

The headline result: Llama3-SWE-RL-70B with Agentless Mini achieves **41.0% pass@1** on SWE-bench Verified, establishing it as the best-performing medium-sized (<100B) open-source model and placing it above GPT-4o with Agentless (38.8%) and comparable to o1-preview with Agentless (41.3%). This result is produced using 500 repair samples per issue and 30 reproduction tests for reranking at temperature 1.0.

Breaking down Table 1 by model category:

- **Against prior open-source methods of comparable size (≤100B):** Llama3-SWE-RL-70B's 41.0% substantially exceeds Lingma-SWE-GPT-72B (28.8%), SWE-Gym-32B (32.0%), SWE-Fixer-72B (32.8%), and even the paper's own SFT baseline Llama3-SWE-SFT-70B (36.2%). The gap between RL and SFT training on the same data is +4.8 percentage points—a 13.3% relative improvement. Critically, all other open-source methods incorporate distilled outputs from GPT-4o or Claude-3.5-Sonnet, while Llama3-SWE-RL uses only publicly available GitHub data with no teacher model.

- **Against proprietary models:** Llama3-SWE-RL-70B outperforms GPT-4o with SWE-agent (23.2%) and Agentless (38.8%), and is roughly tied with o1-preview with Agentless (41.3%). It trails the best proprietary configurations—Claude-3.5-Sonnet with OpenHands (53.0%), Agentless (50.8%), and Tools (49.0%)—by 8–12 percentage points, and trails DeepSeek-R1 (a 671B Mixture-of-Experts model with 37B active parameters, using Agentless at 49.2%) by 8.2 points.

- **Against models using the same scaffold:** Within the Agentless/Agentless Mini scaffold family, the progression from GPT-4o (38.8%) to Llama3-SWE-SFT-70B (36.2%) to Llama3-SWE-RL-70B (41.0%) to DeepSeek-V3 (42.0%) to DeepSeek-R1 (49.2%) to Claude-3.5-Sonnet (50.8%) provides a clear capability ordering. Llama3-SWE-RL-70B sits above GPT-4o and the SFT variant but below the very large open models and the strongest proprietary model. The paper argues this is the best result reported for a medium-sized LLM (<100B) to date.

#### Baseline Comparison: Repair-Only Performance (Table 2)

Table 2 isolates the model's core repair capability by providing oracle localized files in the context, removing localization, test generation, and reranking from the evaluation. The results reveal a stark capability gradient:

- **Base Llama-3.3-70B-Instruct:** With greedy decoding, format accuracy is only 12.2% (most outputs are malformed), yielding a repair performance of 5.4%. With 20-sample majority voting at temperature 0.6 (where malformed outputs are pre-filtered), format accuracy improves to 44.6% and repair performance to 16.6%. This confirms that the base model has some latent repair ability but is severely bottlenecked by its inability to produce correctly formatted search/replace edits.

- **Llama3-SWE-SFT-70B:** Format accuracy jumps to 96.2% and repair performance to 29.6%. SFT is extremely effective at teaching format compliance—nearly all outputs are correctly structured—and provides a substantial (13 percentage point) improvement over the base model's majority-voting repair performance. This establishes that SFT on code-editing data transfers the surface skills of patch generation.

- **Llama3-SWE-RL-70B:** Format accuracy is 95.6% (slightly below SFT) but repair performance is 34.8%—a 5.2 percentage point improvement over the SFT baseline despite slightly worse format compliance. This is the central evidence that RL improves the model's substantive reasoning about bug fixes beyond what SFT achieves: both models produce correctly formatted patches at high rates, but the RL model's patches are correct more often. The gap of 5.2 points on repair-only performance (from 29.6% to 34.8%) represents a 17.6% relative improvement, and this gap is the foundation that the downstream scaffold (localization, test generation, reranking) amplifies to the 4.8-point end-to-end advantage in Table 1.

#### Scaling Analysis with More Repair Samples and Reproduction Tests (Figure 4)

Figure 4 presents two scaling curves showing how the final pass@1 on SWE-bench Verified responds to increasing computational investment in two independent dimensions:

**Left panel: Scaling the number of repair samples (with 30 test samples fixed).**
Starting from 40 repair samples (33.6% pass@1), performance improves rapidly to 160 samples (40.0%), then plateaus with diminishing returns: 40.6% at 320 samples, 41.0% at 500 samples. The curve reveals that 64% of the total gain (from 33.6% to 40.0%, or +6.4 points) is achieved by scaling from 40 to 160 samples, while the remaining 160-to-500 scaling yields only +1.0 additional point. This saturation behavior suggests that for this model, approximately 160–200 repair samples captures most of the available performance, and further scaling provides marginal benefit—likely because additional samples are generating redundant correct patches rather than discovering new correct solutions for previously unsolved instances.

**Right panel: Scaling the number of reproduction tests (with 500 repair samples fixed).**
Using only 1 reproduction test yields 38.8%. Performance improves steadily to 5 tests (39.8%), 10 tests (40.6%), and 20 tests (41.0%), with no further improvement at 30 tests (41.0%). The gain from 1 to 20 tests is +2.2 points, and the saturation at 20 tests indicates that beyond this point, additional reproduction tests do not change reranking outcomes—the dual execution agreement scoring has already converged on stable patch selections. The fact that 1 test already achieves 38.8% (95% of the final performance) suggests that the repair sampling itself, not the sophistication of test-based reranking, is the dominant factor, and reproduction tests primarily help resolve ambiguous cases among highly-ranked patches.

**Key takeaway from Figure 4:** The system's performance can be improved by scaling either the model's generation budget (more candidate patches) or the evaluation budget (more reproduction tests), but generation scaling provides larger absolute gains up to the saturation point. The main evaluation's configuration (500 samples, 30 tests) is at or beyond saturation on both axes, meaning the reported 41.0% is at the asymptotic limit of what this model-scaffold combination can achieve—further improvements would require a better model or a fundamentally different scaffold, not more samples.

#### Generalizability of RL to Out-of-Domain Tasks (Table 3)

Table 3 is arguably the paper's most important experimental result because it tests whether SWE-RL's benefits are confined to the training domain (issue solving) or transfer to other capabilities. The results are organized across five categories:

**Function coding (HumanEval+):** Llama-3.3-70B-Instruct scores 76.2%, Llama3-SWE-SFT-70B drops to 73.2% (−3.0), Llama3-SWE-RL-70B rises to 79.9% (+3.7). The RL model's improvement over the base model on a task it was never trained on is striking: issue-solving RL produces better function-level code generation, while SFT on issue-solving data degrades it.

**Library use (BigCodeBench-Hard):** On the Instruct variant, the pattern is 28.4% (base) → 25.7% (SFT, −2.7) → 28.4% (RL, unchanged). On the Completion variant: 29.1% → 24.3% (SFT, −4.8) → 29.1% (RL, unchanged). The RL model matches the base model's performance exactly on both variants, meaning it neither improves nor degrades this capability. The SFT model loses substantial ground, particularly on the Completion variant.

**Code reasoning (CRUXEval):** On CRUXEval-I (input prediction): 60.5% → 68.4% (SFT, +7.9) → 71.6% (RL, +11.1). On CRUXEval-O (output prediction): 61.9% → 75.1% (SFT, +13.2) → 75.5% (RL, +13.6). This is the one domain where SFT meaningfully improves performance—likely because code execution reasoning is closely related to the code understanding required for bug fixing—but RL still matches or exceeds the SFT gains. The RL model's +11.1 and +13.6 point improvements on CRUXEval are the largest relative gains in the table.

**Math (MATH):** On strict scoring: 63.2% → 54.0% (SFT, −9.2) → 73.7% (RL, +10.5). This is the most dramatic dissociation: SFT causes a severe degradation in mathematical reasoning (nearly 10 points), while RL produces a >10-point improvement. The paper notes that only the RL model consistently follows the "Answer: ..." format requirements for strict scoring, so it also reports lenient scoring: 70.9% → 71.7% (SFT, +0.8) → 73.7% (RL, +2.8). Even under lenient scoring, RL improves over the base model while SFT roughly matches it. The strict-vs-lenient gap for the base model (7.7 points) and SFT model (17.7 points) reveals that both struggle with format compliance on math outputs, while the RL model has no gap (73.7% for both strict and lenient)—suggesting RL has taught the model better instruction-following even in domains where the specific format was never rewarded.

**General language understanding (MMLU):** 86.49% → 85.26% (SFT, −1.23) → 86.82% (RL, +0.33). The effects are smaller in magnitude, consistent with MMLU being a broad knowledge benchmark less affected by domain-specific training. Nonetheless, the pattern holds: SFT slightly degrades, RL slightly improves or preserves.

**Aggregate pattern across all OOD tasks:** The SFT model underperforms the base model on average—it improves on CRUXEval (both variants) and MATH (lenient only) but degrades on HumanEval+, BigCodeBench-Hard (both variants), MATH (strict), and MMLU. The RL model outperforms the base model on 6 of 9 metrics (HumanEval+, CRUXEval-I, CRUXEval-O, MATH strict, MATH lenient, MMLU) and matches on the remaining 3 (BigCodeBench-Hard Instruct and Completion, MATH). In no case does the RL model underperform the base model. The paper frames this as evidence that RL enhances general reasoning (manifesting as improved code reasoning, math, and function coding) while preserving non-reasoning capabilities (manifesting as unchanged library use and language understanding), whereas SFT trades off general capability for task-specific format and content learning.

### Ablation Studies and Robustness Checks

**Reward type: continuous similarity vs. discrete exact-match (Section 3.6, Figure 5):** The paper trains an ablated variant using a discrete 0/1 reward (exact patch match = 1, otherwise = 0) under the same GRPO setup. The continuous reward model achieves 95.6% format accuracy and 34.8% repair (oracle) performance; the discrete reward model achieves 94.2% format accuracy and only 29.0% repair performance. The format accuracy difference is minor (both learn to produce valid patches), but the repair performance gap is 5.8 points—nearly identical to the gap between SFT and RL in Table 2, suggesting that continuous reward is roughly as important as the RL vs. SFT distinction itself. The training dynamics plot (Figure 5 right) shows why: the discrete reward averages near zero throughout training (the model rarely generates exact matches to the oracle patches, even after 1,600 steps), while the continuous reward shows monotonic improvement from below 0.2 to above 0.4. The discrete reward provides no gradient to guide improvement—a patch that fixes the bug but uses different variable names receives zero reward, indistinguishable from a patch that deletes all code. The continuous reward credits partial progress, enabling the optimization to climb the reward landscape.

**Format penalty (-1 vs. hypothetical 0):** While not ablating the value directly, the paper reports that the base model achieves only 12.2% format accuracy under greedy decoding (Table 2), and that both SFT (96.2%) and RL (95.6%) achieve high format accuracy after training. The −1 penalty for format errors in the RL reward (versus a counterfactual 0 penalty) is credited with teaching format compliance: a malformed patch is strictly worse than a perfectly formatted but content-free patch (which would score 0 similarity), creating a strong gradient toward proper formatting. The fact that RL achieves 95.6% format accuracy—comparable to SFT's 96.2%—without any explicit supervised format training indicates the penalty is effective.

**SFT baseline as an ablation of training paradigm (Tables 2 and 3):** The SFT baseline represents the most comprehensive ablation in the paper—it tests whether the benefits of SWE-RL derive from RL specifically or from simply training on more software engineering data. Using the same seed PRs as SWE-RL but processed through synthetic data generation and SFT, the baseline achieves 36.2% end-to-end (vs. RL's 41.0%) and 29.6% repair-only (vs. RL's 34.8%). On OOD tasks, SFT shows average degradation while RL shows average improvement. This three-way comparison (base model → SFT → RL) isolates the effect of RL from the effect of domain-specific training. The paper argues the difference arises because SFT teaches output distributions while RL teaches reasoning processes, with the KL penalty in GRPO preventing the capability loss that SFT's distribution-matching objective induces.

**Effect of hardware scale / training duration:** The paper does not ablate training steps, batch size, number of GPUs, or model size, making it unclear whether the 1,600-step, 512-GPU configuration is near convergence or whether further training would yield additional improvements. The training reward curve for continuous reward (Figure 5) continues rising at step 1,600, suggesting additional training might help, but no longer runs are reported. Similarly, no smaller-model ablations are conducted to test whether SWE-RL's benefits scale with model size or would work on 7B/13B variants.

**Effect of data curation choices:** The paper does not ablate the merge-base retrieval, relevant-files prediction, or filter thresholds in the data curation pipeline. The claim that these choices are important for training quality (Section 3.4) is supported by qualitative reasoning and reference to SWE-bench fine-tuning experiments (Jimenez et al., 2023) but not by direct ablation evidence in this paper. The effect of the 273k seed selection from the 11M raw PRs is also not studied—would training on all 11M help (more data) or hurt (more noise)?

**Effect of Agentless Mini scaffold simplifications:** The paper does not compare Agentless Mini against the original Agentless scaffold with the same model, making it difficult to assess whether the scaffold simplifications (file-level-only localization, multiple reproduction tests) help or hurt relative to the standard Agentless configuration. The paper's reported baselines for GPT-4o and Claude-3.5-Sonnet use the original Agentless, not Agentless Mini, so the scaffold difference is a confound in interpreting the Table 1 comparisons.

**Generalization to untrained scaffold steps:** The paper claims that Llama3-SWE-RL generalizes to file-level localization and test generation despite being trained only on repair (Section 2, Section 3.1, Appendix B). This claim is implicit in the end-to-end pipeline working—since the model generates localization candidates and test samples, it must be capable of these tasks—but no ablation isolates the model's localization accuracy or test generation quality independently. It is possible that the scaffold's localization and test generation steps succeed *despite* mediocre model outputs (e.g., by generating many samples and filtering), and that the RL training did not actually improve these capabilities. A direct comparison of localization accuracy between the base model, SFT model, and RL model—independent of the full pipeline—would strengthen this claim.

**Reproduction test scaling:** The right panel of Figure 4 shows saturation at 20 tests, but does not ablate the quality of the reproduction tests. It is possible that better reproduction tests (generated by a different model or selected with different criteria) would continue to scale beyond 20, while poorly generated tests saturate early because they don't discriminate among patches.

**Out-of-domain vs. in-domain tension:** The OOD results (Table 3) use zero-shot greedy decoding on standard benchmarks, while the in-domain results (Table 1) use 500 samples with temperature 1.0 and complex reranking. The gap between these evaluation protocols makes it unclear how much of the OOD improvement is due to general reasoning enhancement vs. improved instruction-following and format compliance (as suggested by the MATH strict-vs-lenient gap). A repair-only evaluation under the OOD protocol (zero-shot greedy) would help calibrate: does RL improve repair at zero-shot, or only when amplified by 500× sampling?

### Critical Assessment

#### Claim 1: "Llama3-SWE-RL-70B achieves 41.0% on SWE-bench Verified, the best performance among medium-sized language models (<100B) and even comparable to leading proprietary models like GPT-4o."

This claim is well-supported by Table 1, subject to two important contextualizations. First, the "comparable to GPT-4o" framing is scaffold-dependent: GPT-4o with Agentless achieves 38.8% (Table 1), so Llama3-SWE-RL-70B is 2.2 points ahead under the Agentless Mini/Agentless scaffold comparison—but this compares a model explicitly RL-trained for the task against a proprietary model used off-the-shelf (no task-specific training). GPT-4o might close or reverse the gap with scaffold optimization or with its own RL training on software engineering data. Second, the "best among <100B" claim is true at time of writing against the baselines listed, but the comparison set is small—only 5 other open-source models in this size class have published SWE-bench Verified results, and all are from different research groups with different training recipes and scaffolds. The claim would be stronger with additional head-to-head comparisons using identical scaffolds and evaluation protocols. The paper acknowledges this limitation implicitly by reporting both their own SFT baseline and aggregate results, but does not control for scaffold differences across the compared methods.

#### Claim 2: "SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data."

This claim contains two sub-claims. The first—that the model learns to solve issues from PR data—is directly demonstrated by the repair performance improvement from 5.4% (base model greedy) to 34.8% (RL model greedy, Table 2). The second—that the model "recovers a developer's reasoning processes"—is a stronger claim about *how* the model solves issues, not just that it does. The qualitative examples in Figure 3 show the model engaging in self-reflection and debugging-like reasoning, but these are cherry-picked examples from a model that solves 41.0% of issues end-to-end. It is unknown whether the model's reasoning traces on unsolved issues are coherent (merely arriving at wrong conclusions) or nonsensical (formally correct reasoning but disconnected from the actual bug). This is a genuine gap: the paper provides no systematic evaluation of reasoning quality (e.g., human evaluation of thinking blocks, correlation between reasoning coherence and patch correctness, or comparison of reasoning traces between RL and SFT models). The "recover" framing is suggestive but not rigorously tested.

#### Claim 3: "Despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills... whereas a supervised-finetuning baseline even leads to performance degradation on average."

This is the paper's strongest and most convincingly supported claim. Table 3 provides a clean nine-metric comparison where RL improves over the base model in 6 metrics, matches in 3, and degrades in none, while SFT improves in only 3 metrics (CRUXEval-I, CRUXEval-O, MATH lenient) and degrades in 5 (HumanEval+, BigCodeBench-Hard both variants, MATH strict, MMLU). The effect is consistent across diverse task categories (code generation, code reasoning, math, language understanding) and survives the lenient-vs-strict scoring variation on MATH. The statistical significance analysis (≥0.05 aggregate significance across benchmarks, per Eval Arena thresholds) addresses the concern that small absolute gains on individual benchmarks might be noise.

However, a limitation worth noting: the SFT baseline degrades on OOD tasks despite being trained on a "meticulously curated data mix" of general coding and dialog data, but the paper doesn't provide the exact mixture ratios or ablate them. It is possible that a different mixture (e.g., more general data, less code-editing data) would prevent the SFT degradation while still capturing some of the in-domain improvement. The claim "SFT leads to performance degradation" is true for this specific SFT recipe, but might not hold for a more carefully balanced SFT mixture. The RL model, by contrast, achieves OOD preservation "for free" through the KL penalty mechanism, without requiring mixture tuning—this is a practical advantage of the RL approach even if SFT could theoretically match it with enough mixture engineering.

Additionally, the OOD evaluation uses zero-shot greedy decoding (Table 3) while the in-domain evaluation uses 500 samples with reranking (Table 1). The RL model's OOD improvement is demonstrated under a protocol closer to how the base model was originally evaluated (zero-shot), which strengthens the claim that RL genuinely enhanced the model's capabilities rather than merely its test-time compute scaling. But the gap between evaluation protocols means we cannot directly compare the magnitude of in-domain and OOD improvement on a common scale—the 41.0% in-domain number reflects aggressive sampling and reranking, while the 79.9% on HumanEval+ reflects a single greedy generation.

#### Claim 4: SWE-RL is "the first approach to scale RL-based LLM reasoning for real-world software engineering."

This is a scope claim about novelty. The paper cites DeepSeek-R1's limited SE results and prior work's reliance on SFT+distillation, establishing that RL with rule-based rewards had not been successfully applied to SWE-bench-style tasks before. Within the paper's framing—RL with rule-based similarity rewards on open-source PR data—this claim appears accurate. However, concurrent or unpublished work might weaken the "first" claim over time, and the claim's strength depends on what "real-world software engineering" includes—RL has been applied to program repair (Gehring et al., 2025) and competitive programming (DeepSeek-AI, 2025), which are software engineering-adjacent, but not to repository-level issue solving with multi-file patches. The paper's specific contribution—continuous similarity reward enabling RL on real patches without execution or exact match—is genuinely novel.

#### Genuine weaknesses and missing experiments:

- **No evaluation on non-Python repositories or non-SWE-bench tasks:** All results are on Python repositories from SWE-bench. The RL training data is multi-language (drawn from all of GitHub), but evaluation is Python-only. It's unknown whether the model's improvements transfer to other languages or to software engineering tasks beyond issue fixing (e.g., code review, refactoring, documentation generation).

- **No direct measurement of localization or test generation improvement:** The claim that RL generalizes to these untrained scaffold steps is inferred from end-to-end performance, not directly tested. An ablation evaluating localization accuracy (Top-K file retrieval) and test generation quality (fraction of generated tests that correctly reproduce and resolve the issue) for base, SFT, and RL models would substantially strengthen this claim.

- **The 41.0% result depends on 500 repair samples and 30 reproduction tests:** The scaling analysis (Figure 4) shows diminishing returns, but the compute cost of this evaluation protocol is not quantified. A smaller, more efficient configuration (e.g., 160 samples, 20 tests) achieves 40.0-41.0%, but the paper only reports this in the scaling plot, not as a recommended deployment configuration. For practitioners, knowing that 41.0% requires 500× generation cost while 40.0% requires ~160× would be valuable.

- **No model scale ablations:** All experiments use 70B models. Whether SWE-RL's benefits scale down to 7B/13B (more practical for many researchers) or up to larger models is unknown. The paper also doesn't compare against simply using a larger base model with greedy decoding—would Llama-3.1-405B-Instruct with zero-shot greedy repair outperform Llama3-SWE-RL-70B with 500 samples?

- **No data scale ablations:** The 273k seed PRs are a curated subset of 11M raw PRs. It's unknown whether training on the full 11M (with more noise) or an even more aggressively filtered subset would change results. The paper's data pipeline is complex (Figure 6) with many filtering heuristics, but none are empirically validated through ablation.

- **Limited analysis of failure modes:** The paper reports success rates but never analyzes what the model gets wrong on the remaining 59% of SWE-bench Verified instances. Are failures due to localization errors, repair errors, test generation errors, or execution environment issues? This breakdown would clarify where future work should focus and whether the RL model's repair capability (34.8% with oracle files, Table 2) is the primary bottleneck or whether the pipeline stages introduce most of the failures.

- **Comparison of training paradigms uses different data processing pipelines:** The RL model trains on raw (issue, context, oracle patch) triples. The SFT model trains on synthetically generated chain-of-thought + edit demonstrations, plus additional general SFT data. The SFT model sees chain-of-thought examples during training; the RL model discovers its own thinking strategies. This means the RL-vs-SFT comparison conflates the training objective (RL vs. SFT) with the data format (raw patches vs. synthetic demonstrations) and the presence/absence of teacher-generated reasoning traces. A cleaner ablation would train an SFT model on the raw PR data (issue → patch, possibly with human-written PR descriptions as chain-of-thought proxies) to isolate the objective function's effect.

- **The "aha moments" (Figure 3) are illustrative, not systematically analyzed:** The paper shows 3 qualitative examples of emergent reasoning, but doesn't quantify how often such patterns occur, whether they correlate with correct solutions, or whether they emerge at a particular point in training. A quantitative analysis—e.g., measuring the frequency of self-reflection keywords ("wait," "however," "alternatively") over training steps, or comparing thinking block length between correct and incorrect solutions—would transform the qualitative observation into a verifiable claim about RL's effect on reasoning.

Overall, the paper's central contributions are well-supported by the experimental evidence within the scope of what was tested. The 41.0% SWE-bench Verified result is genuine and reproducible (code and evaluation pipeline are released). The RL-vs-SFT generalization pattern in Table 3 is compelling and likely to be the most influential finding. The main weaknesses are the lack of systematic analysis of what the model actually learns (reasoning quality, localization accuracy, failure modes), the single-scale single-scaffold evaluation, and the confound between training objective and data format in the RL-vs-SFT comparison. These are not fatal—they represent productive directions for follow-up work—but they mean the paper's claims about *why* RL works (process reasoning vs. output mimicking) remain hypotheses supported by outcome evidence rather than direct mechanistic tests.

## 6. Limitations and Trade-offs

### The Reward Signal Prizes Textual Similarity, Not Semantic Correctness

The central enabling mechanism of SWE-RL—the `difflib.SequenceMatcher` similarity between predicted and oracle patches—is simultaneously its greatest strength and its most fundamental limitation. The reward function defined in Equation 1 rewards the model for producing patches that *look like* what the human developer wrote, not patches that *are functionally equivalent* to what the human developer wrote. The paper explicitly acknowledges this in Section 5 (Limitations):

> "our reward implementation compares the sequence similarity between the predicted and oracle patch rather than their semantic equivalence. This may prevent the policy LLM from exploring alternative, functional equivalent solutions."

**The consequence.** This creates a hard upper bound on the model's potential: it can never exceed the oracle patch's quality, and it is actively penalized for discovering better or alternative fixes. Consider a scenario where the human developer's fix introduces a subtle bug that was never caught—the model would be rewarded for reproducing that bug, not for avoiding it. More practically, for any given bug, there may exist dozens of functionally equivalent patches (different variable names, different control flow structures, different error handling strategies) that are all correct, but only the exact merged PR patch receives maximal reward. The model is being optimized to mimic one specific developer's coding style, not to solve the underlying problem. The paper's reward ablation (Figure 5) shows that the discrete exact-match variant collapses to near-zero reward throughout training, but even the continuous variant is ultimately measuring textual overlap, not behavioral correctness. A patch that deletes the buggy function entirely and replaces it with a call to a correct helper might be functionally perfect but textually dissimilar, receiving a low reward.

**What evidence exists in the paper.** The reward ablation in Section 3.6 and Figure 5 is the primary evidence that the specific reward formulation matters enormously. The discrete variant (29.0% repair, near-zero training reward) vs. continuous variant (34.8% repair, rising reward) shows that the model is highly sensitive to how "similarity" is defined. However, this ablation only explores the continuous-vs-discrete axis within sequence matching—it does not test against a semantic reward (e.g., execution-based, or AST-diff-based), so it does not quantify how much performance is left on the table by the sequence similarity choice. The paper provides no analysis of false positives (patches that score high similarity but are functionally wrong) or false negatives (patches that score low similarity but are functionally correct), which would directly characterize this limitation's severity.

**Mitigation status.** The paper acknowledges this limitation explicitly but does not attempt to address it within the current framework. The authors suggest no concrete mitigation beyond flagging it as a limitation. A natural mitigation—training a verifier model to assess semantic equivalence and using its output as a reward signal, analogous to how RLHF trains reward models—is not explored. The reliance on oracle patches as the sole source of "correctness" signal means the approach is fundamentally bounded by the quality and uniqueness of human-written patches. For repositories where the merged PR contains suboptimal or bug-introducing fixes (which is not uncommon in real-world development), the RL training would actively teach the model to reproduce those flaws.

### Difficulty Estimation Cost Is Completely Unaccounted For

The main results in Table 1 (41.0% on SWE-bench Verified) are achieved using **500 repair samples per issue** with **30 reproduction tests** at temperature 1.0, with the best patch selected via a complex reranking procedure involving regression test execution and dual execution agreement scoring. The repair-only evaluation (Table 2) uses a single greedy sample and achieves only 34.8%. The gap between these numbers—6.2 percentage points, or an 18% relative improvement—comes entirely from massive oversampling and test-based filtering at inference time. This means the headline 41.0% number is not a measure of the model's ability to produce a correct patch in a single attempt; it is a measure of the model's ability to produce a correct patch *somewhere in a set of 500 independent attempts*, combined with the scaffold's ability to identify it.

**The consequence.** The computational cost of evaluation is not reported, but it is enormous. For each of 500 SWE-bench Verified issues, the pipeline: (1) generates 500 repair samples from a 70B model (each with a chain-of-thought reasoning trace), (2) generates up to 30 reproduction tests, (3) executes all regression tests against each candidate patch, (4) executes all reproduction tests against each surviving patch, and (5) computes the dual execution agreement scores. The generation cost alone is 500 × 500 = 250,000 forward passes of a 70B model just for repair, plus additional passes for localization and test generation. This makes the evaluation protocol completely impractical for any latency-sensitive or cost-sensitive deployment. A practitioner who reads "41.0% on SWE-bench Verified" might reasonably assume this reflects the model's single-attempt accuracy; it does not. It reflects what happens when you are willing to spend roughly **500× more inference compute** than a single forward pass, plus execution costs. The scaling analysis in Figure 4 (left) shows that performance at 40 repair samples is only 33.6%—a 7.4 point gap from the 500-sample result—suggesting that the majority of the headline performance depends on the sampling budget.

The paper never reports the single-attempt pass@1 of Llama3-SWE-RL-70B on the full SWE-bench Verified pipeline (the 34.8% in Table 2 is repair-only with oracle file localization). For a fair comparison against models that use standard inference budgets (e.g., GPT-4o with Agentless at 38.8%, which presumably generates far fewer than 500 samples per issue), the single-sample or low-sample performance would be the appropriate metric. The 41.0% vs. 38.8% comparison is comparing a 500-sample RL model against a (likely) single-digit-sample proprietary model, which makes the RL model's advantage appear larger than it would be under a matched inference budget.

**What evidence exists in the paper.** The scaling analysis in Figure 4 directly quantifies this dependence. At 40 samples: 33.6%. At 160 samples: 40.0%. At 500 samples: 41.0%. The curve is largely saturated by 160 samples, but 160× is still far beyond standard inference. Table 2 provides the single-sample repair-only number (34.8%), but only with oracle localization—which removes the file-finding step that the full pipeline requires. The paper does not report single-sample end-to-end performance with Agentless Mini. The training compute is reported (512 H100 GPUs, 32 hours), but the inference compute for evaluation is never quantified.

**Mitigation status.** Not addressed. The paper treats the 500-sample budget as a fixed evaluation protocol and investigates how performance scales with it (Figure 4), but never discusses the practical implications of requiring such a large sampling budget for the headline result. The paper does not propose more efficient inference strategies (e.g., early stopping when a consensus group emerges, adaptive sampling based on problem difficulty, or distillation of the 500-sample behavior into a single-sample model). The "Limitations" section in the paper focuses on the reward function and scaffold design, not on the inference cost of the evaluation protocol.

### The SFT Baseline Conflates Training Objective with Data Format and Scale

The paper's central comparative claim—that RL leads to generalized reasoning improvements while SFT leads to task-specific overfitting—rests on a single SFT baseline, Llama3-SWE-SFT-70B. The construction of this baseline, described in Section 3.1 and Appendix C, introduces several confounds that weaken the RL-vs-SFT comparison:

1. **Different data generation pipelines:** The RL model trains on raw (issue, context, oracle patch) triples with no intermediate processing. The SFT model trains on synthetic data generated by Llama-3.3-70B-Instruct in a Magicoder-style pipeline: the base model is prompted with the ground-truth PR and oracle patch to generate chain-of-thought reasoning traces and edit demonstrations, which are then filtered against the ground truth. This means the SFT model is learning from a *teacher model's reasoning patterns*, not from the raw PR data directly. If the teacher model produces flawed or superficial reasoning (as is common with LLM-generated chain-of-thought), the SFT model inherits those flaws. The RL model, by contrast, discovers its own reasoning strategies.

2. **Different training data mixtures:** The SFT model is trained on a mixture of synthetic code editing data, Llama 3 coding SFT data, and Llama 3 general SFT data, for a total of 2 billion tokens. The RL model is trained on only the seed PR dataset. The SFT model thus sees substantially more data and a more diverse data distribution, including general dialog and coding data that should *help* preserve OOD performance. Yet it still degrades. This actually strengthens the paper's claim (SFT degrades despite more diverse data), but it means the comparison is not a clean test of objective function (RL vs. SFT)—it also tests data mixture, data generation method, and total training tokens.

3. **Presence/absence of chain-of-thought supervision:** The SFT model is explicitly trained on chain-of-thought traces (the synthetic data includes thought processes). The RL model receives no chain-of-thought supervision—it discovers reasoning strategies purely through outcome rewards on patches. The "aha moments" in Figure 3 are claimed as emergent properties of RL, but the comparison model (SFT) was never given the opportunity to develop emergent reasoning because it was spoon-fed reasoning traces during training. A fairer comparison would include an SFT model trained on raw PR data without synthetic reasoning traces, or an RL model initialized with chain-of-thought supervision.

**The consequence.** The claim that "RL generalizes while SFT overfits" could be rephrased more precisely as: "The specific RL pipeline described in this paper generalizes better than the specific SFT pipeline described in this paper, when both are applied to Llama-3.3-70B-Instruct." This is a weaker but more accurate claim. The confounds mean we cannot attribute the generalization difference to RL vs. SFT alone—it could be due to the synthetic reasoning traces introducing harmful patterns, the data mixture ratios being suboptimal, or the SFT model overfitting to the specific formatting and style of the teacher-generated data rather than to the task itself. The paper's failure to ablate these confounds (e.g., by training an SFT model on raw PR data with human-written issue descriptions as chain-of-thought, or by varying the SFT data mixture) leaves the core mechanism of generalization unexplained.

**What evidence exists in the paper.** Table 3 provides the evidence for the generalization claim, and Table 2 provides the in-domain repair comparison. The data generation differences are described in Appendix C. The paper does not present any ablation that varies the SFT data mixture, the source of synthetic reasoning traces, or the inclusion/exclusion of chain-of-thought supervision. The only SFT variant tested is the one described in Section 3.1.

**Mitigation status.** Not addressed directly. The paper presents the SFT baseline as a carefully constructed strong baseline ("meticulously curated data mix"), implying that the comparison is fair. However, the differences in data pipeline and training recipe are treated as implementation details rather than as potential confounds requiring ablation. The paper does not discuss this limitation or propose future experiments to disentangle the objective function effect from the data format effect.

### Generalization Is Demonstrated on Only Five Benchmarks, All in English, All Text-Based

The paper's most celebrated finding—that RL on software evolution data produces generalized reasoning improvements—is demonstrated on five out-of-domain benchmarks: HumanEval+, BigCodeBench-Hard, CRUXEval, MATH, and MMLU (Table 3). While this is a reasonable initial evaluation, the scope of "generalization" being claimed is much narrower than the term implies.

**The consequence.** The benchmarks share important characteristics with the training domain that may inflate the apparent generalization:

- **All are text-in/text-out tasks** that structurally resemble the training format (read a prompt, produce a text response). The model was not tested on tasks requiring different output modalities (e.g., code execution traces, structured JSON outputs, multi-turn interactions).

- **All are in English**, while GitHub PRs span dozens of natural languages in issue descriptions and comments. If the RL training biased the model toward English technical reasoning, the benchmark suite would not detect this.

- **HumanEval+, BigCodeBench-Hard, and CRUXEval are all Python-centric**, matching the language of SWE-bench (which is Python-only) and likely the dominant language in the training data (GitHub is Python-heavy). The paper provides no evidence that improvements transfer to other programming languages, despite the training data being multi-language.

- **MMLU is a multiple-choice benchmark** where small absolute gains (86.49% → 86.82%, +0.33) could reflect improved instruction-following (better adherence to the multiple-choice format) rather than improved factual knowledge. The paper's own analysis of MATH strict vs. lenient scoring shows that format compliance is a confound in at least one benchmark—the base model drops 7.7 points under strict scoring, the SFT model drops 17.7 points, but the RL model drops 0 points. This improvement could be entirely due to RL teaching better format adherence, not better math.

- **No tasks requiring factual recall, common sense reasoning, or world knowledge** beyond MMLU, which is a broad but shallow knowledge test. The paper does not test whether RL on code data preserves or degrades the model's ability to answer questions about history, literature, or science.

The claim that the model has developed "generalized reasoning skills" (abstract, Section 2.2) overstates what the evidence supports. It would be more accurate to say: the model shows improved performance on several code-adjacent and mathematical reasoning benchmarks, with no detected regression on the tested set.

**What evidence exists in the paper.** Table 3 is the sole evidence. The benchmarks were chosen deliberately to span function coding, library use, code reasoning, math, and general language understanding, which is a reasonable diversity claim. However, the paper does not discuss the shared characteristics of these benchmarks with the training domain or argue why they constitute a sufficient test of generalization.

**Mitigation status.** Not addressed. The paper treats the five benchmarks as a comprehensive test of generalization and does not acknowledge the scope limitation. Future work suggested in Section 5 focuses on improving the RL approach and scaffold, not on broader generalization testing.

### All Experiments Use a Single Model Family at a Single Scale

Every result in the paper—the 41.0% SWE-bench Verified score, the RL-vs-SFT comparison, the reward ablation, the scaling analysis, the OOD generalization—is produced using exactly one base model (Llama-3.3-70B-Instruct) at exactly one scale (70B parameters). The paper provides no evidence that SWE-RL works with other model families (e.g., Qwen, DeepSeek, Mistral), other model sizes (7B, 13B, 405B), or even other variants within the Llama family (e.g., the base model vs. the instruct-tuned variant).

**The consequence.** This is not a minor omission—it means we have no idea whether SWE-RL's core mechanisms are general properties of RL on software data or idiosyncratic properties of Llama-3.3-70B-Instruct's specific pretraining. Several mechanisms could be model-specific:

- The emergent "aha moments" (Figure 3) depend on the base model's ability to produce coherent chain-of-thought reasoning in the first place. A weaker base model might never develop self-reflection because its thinking traces are too incoherent for RL to shape. A stronger base model might already exhibit these behaviors without RL. The paper provides no evidence about where in the capability spectrum SWE-RL's benefits emerge.

- The 95.6% format accuracy achieved through RL (Table 2) depends on the base model's instruction-following ability—Llama-3.3-70B-Instruct is specifically trained for instruction following. A base model without instruction tuning might never learn the search/replace format through RL alone, because the −1 format penalty provides only a negative signal (penalizing wrong formats) without positive demonstrations of correct formats. The base model's 12.2% format accuracy (Table 2) shows that even the instruct-tuned variant struggles with format pre-RL; a non-instruct model might find the format reward landscape entirely unnavigable.

- The OOD generalization pattern (Table 3) might depend on Llama-3.3-70B-Instruct's specific pretraining data mixture. If the model's general capabilities are unusually robust to RL fine-tuning because of how it was pretrained, the generalization result might not replicate with other models.

- The 41.0% result is achieved with Agentless Mini, a scaffold designed specifically for this model and training pipeline. It's unknown whether the same scaffold works with other RL-trained models, or whether other scaffolds would produce different results with Llama3-SWE-RL-70B.

**What evidence exists in the paper.** None. The paper does not include any experiments varying the base model, model scale, or model family. The related work section discusses other models (Qwen2.5-Coder, DeepSeek-Coder, SWE-Llama) but only as prior work baselines, not as alternative base models for SWE-RL. The training configuration (Section 3.1) specifies Llama-3.3-70B-Instruct as the sole initialization.

**Mitigation status.** Not addressed. The paper does not acknowledge this as a limitation or discuss the generalizability of SWE-RL across model families. Given that the paper's primary contribution is a training methodology (not a specific model), the lack of cross-model validation is a significant gap in the evidence for the methodology's generality. The title claims "SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution"—an unqualified claim about the methodology—but the evidence supports only "SWE-RL advances Llama-3.3-70B-Instruct's reasoning."

### The Pipeline-Based Scaffold Prevents the Model from Learning Interactive Debugging Skills

SWE-RL's training design deliberately simplifies the model's task to a single-turn generation: receive issue + full file contents → produce a patch. The Agentless Mini scaffold (Appendix B) then decomposes SWE-bench evaluation into separate stages (localization, repair, test generation, test selection, reranking) that run as independent inference calls with no feedback loop. The paper frames this simplicity as a strength—it makes RL training tractable and forces the model to develop internal fault localization reasoning. However, this design choice also imposes a fundamental limitation on what the model can learn and how the system can recover from errors.

**The consequence.** In real-world software engineering, debugging is inherently interactive and iterative. A developer does not read the entire codebase, think for an hour, and produce a perfect patch in one shot. They form hypotheses, inspect code, run tests, observe failures, refine their understanding, and iterate. The Agentless Mini pipeline eliminates this feedback loop completely:

- If the localization stage (Stage 1) selects the wrong files, the repair stage has no mechanism to detect this or request different files. It must produce a patch based on irrelevant context, which will almost certainly be wrong or inapplicable.
- If the repair stage produces a patch that introduces a regression, the model never sees the regression test failures and cannot revise its patch. The reranking stage (Stage 4) can filter out regression-introducing patches, but only from the pool of already-generated candidates—the system cannot ask the model to "try again, avoiding this specific regression."
- The reproduction test generation stage (Stage 2) operates independently of the repair stage. The model cannot use information from the repair attempt to generate better tests, or use test execution results to refine its understanding of the bug.

The paper acknowledges this limitation explicitly in Section 5:

> "as a pipeline-based approach, Agentless Mini divides all steps into distinct inference stages. This 'external structure' prevents the model from learning through interaction feedback and hinders its ability to consider the entire problem holistically."

This limitation is especially significant because SWE-bench's most successful approaches are agentic—Claude-3.5-Sonnet with OpenHands (53.0%) and Tools (49.0%) both use interactive scaffolds where the model can explore the repository, run commands, observe outputs, and adjust its approach. The 8–12 point gap between Llama3-SWE-RL-70B (41.0%) and these agentic systems may partly reflect the fundamental ceiling of a single-pass pipeline architecture: some bugs simply cannot be diagnosed or fixed from static code context alone, no matter how strong the model's reasoning.

**What evidence exists in the paper.** The paper does not directly measure how much performance is lost due to the non-interactive scaffold. However, the repair-only evaluation in Table 2 provides an upper bound on what the pipeline can achieve if the other stages were perfect: 34.8% with oracle file localization and greedy decoding. The end-to-end result (41.0%) is higher because of 500× oversampling and test-based reranking, which partially compensates for errors in localization and repair by generating many candidates and filtering. But the fundamental limitation remains: the model cannot recover from localization errors, and it cannot use execution feedback to refine its patches. An ablation comparing Agentless Mini against an agentic variant using the same RL model (e.g., giving the model access to a bash tool and letting it iterate) would quantify this limitation but is not performed.

**Mitigation status.** The paper acknowledges this limitation in the Limitations section but does not attempt to address it. The authors do not propose a hybrid approach (e.g., RL training on single-turn repair followed by RL fine-tuning in an interactive environment) or discuss whether the single-turn RL training would transfer to an agentic setting. The choice of a pipeline-based scaffold is presented as a deliberate simplification to make RL training feasible, but the paper does not explore whether this simplification is temporary (a first step toward interactive RL) or permanent (the authors believe single-turn repair is sufficient for most issues). Given that agentic approaches dominate the SWE-bench leaderboard, this is a consequential trade-off that deserves more discussion than a brief acknowledgment in the limitations paragraph.

## 7. Implications and Future Directions

### How This Work Changes the Landscape

This paper introduces a **methodological reframing** rather than an incremental improvement: it demonstrates that real-world software engineering data—the messy, multi-file, non-executable record of GitHub pull requests—can serve as a self-contained reinforcement learning environment, and that the key enabling mechanism is a continuous similarity reward that provides a learning gradient where exact-match and execution-based rewards fail. This is not a paradigm shift in the sense of upending existing approaches (the dominant paradigm of distillation from proprietary models remains viable), but it opens a **parallel research track** that was previously considered infeasible: training open models to solve real software issues without teacher models, without execution environments, and with only publicly available data.

**The most important reframing: RL on SE data is now an alternative to distillation, not a complement to it.** Prior to SWE-RL, every open-source model achieving non-trivial SWE-bench performance relied on some form of distillation from GPT-4o or Claude-3.5-Sonnet (Lingma-SWE-GPT, SWE-Gym, SWE-Fixer, as documented in Table 1). The implicit assumption was that open models could not autonomously learn the complex reasoning required for repository-level bug fixing from raw data alone—they needed a stronger teacher to demonstrate the target behavior. SWE-RL falsifies this assumption: the 41.0% achieve rate on SWE-bench Verified, produced without any teacher model, is not only competitive with distillation-based approaches in the same size class (beating Lingma-SWE-GPT-72B's 28.8% and SWE-Fixer-72B's 32.8%) but actually exceeds them. This means the ceiling on open-source software engineering models is no longer bounded by access to proprietary systems—it is bounded by the quality of publicly available software evolution data, the design of reward functions, and the scale of RL training.

**The reward design insight generalizes beyond software engineering.** The paper's central technical contribution—that continuous sequence similarity can substitute for discrete correctness verification when exact matching is too sparse—is not specific to code patches. Any domain where the target output is a structured artifact with meaningful intermediate similarity to incorrect attempts could adopt this approach: document editing (comparing predicted and ground-truth document diffs), configuration management (comparing system configuration changes), spreadsheet manipulation (comparing predicted and correct spreadsheet formulas), or even structured data extraction (comparing predicted and ground-truth JSON/XML outputs). The finding that the discrete reward variant plateaus at near-zero reward while continuous reward provides steady improvement (Figure 5) is a diagnostic that will generalize: whenever the space of correct outputs is large and diverse, smoothing the reward landscape through partial credit is necessary for RL to make progress.

**It resolves the tension between "RL works for coding" (DeepSeek-R1) and "RL for SE is limited" (DeepSeek-R1 report).** The DeepSeek-R1 technical report noted limited effectiveness on software engineering tasks, which might have led the field to conclude that RL is not well-suited to real-world SE. SWE-RL identifies the specific reason for this limitation—the reward signal, not the RL algorithm or the task domain—and provides a fix. The deep lesson is that the DeepSeek-R1 paradigm (RL with rule-based rewards) transfers to messy real-world domains, but only if the reward function is adapted to the domain's specific verification challenges. Exact match works for math because answers are short and canonical. Execution works for competitive programming because problems come with test harnesses. Sequence similarity works for real-world patches because it captures partial progress in a way that these other signals cannot. Each domain requires its own reward engineering, and the paper provides a template for how to think about that engineering: identify what makes the domain's correct outputs diverse, and design a continuous signal that credits partial alignment.

**The RL-vs-SFT generalization dissociation (Table 3) changes how the field should think about incorporating domain data.** The dominant approach to improving LLMs on specialized tasks is SFT on task-specific demonstrations, often with a small amount of general data mixed in to "prevent catastrophic forgetting." SWE-RL's results suggest this approach may be fundamentally misguided: SFT degrades OOD performance even when a carefully curated data mixture is used, while RL preserves or improves OOD performance automatically through the KL penalty mechanism. If this dissociation replicates across other domains (medicine, law, finance, science), it would motivate a shift from "SFT with general data mixture" to "RL with KL constraint" as the default recipe for domain adaptation. This is not a minor methodological preference—it is a claim about the inductive biases of different training objectives that, if true, has consequences for every organization fine-tuning LLMs on proprietary domain data.

**It establishes the merge-base + distractor file strategy as a curriculum design principle.** The paper's data curation decisions—retrieving code at the merge base rather than the main branch head, including relevant-but-unchanged files as negative examples—were presented as engineering details but embody a general principle: the context you provide and the reward you compute jointly define what the model learns, and you can teach composite skills (localization + repair) through a single outcome reward by carefully constructing the input to require those skills. This principle will influence how future work designs RL environments for complex tasks: rather than decomposing the task into separately trained subtasks, structure the input so that the model must perform the subtasks as internal reasoning to maximize the outcome reward, and let the optimization discover the necessary subgoals.

**Research directions that become more attractive:**
- **Reward function engineering for diverse output domains.** The paper's reward ablation (Figure 5) shows that the specific form of the reward function is a first-order determinant of RL success. This opens a research area: cataloging which types of continuous similarity signals work for which types of structured outputs, and developing principles for reward design that go beyond the "try sequence matcher and hope it works" approach used here.
- **Scaling laws for RL on software data.** The paper uses 273k seed PRs and 1,600 training steps but performs no data or compute scaling ablations. Understanding how performance scales with dataset size, training steps, and model size would place SWE-RL in the same framework as pretraining scaling laws, enabling resource allocation decisions.
- **Hybrid SFT+RL pipelines.** The paper compares SFT and RL as alternatives, but the most performant recipe might combine them: SFT to teach format compliance and basic editing patterns, followed by RL to optimize for patch quality. The 95.6% format accuracy of the RL model (Table 2) was achieved through RL alone, but SFT achieves 96.2% while teaching format much faster (SFT on 2B tokens vs. RL over 1,600 steps of exploration). An SFT warmup phase could accelerate RL training by initializing the model in a region of the policy space where it already produces valid patches, allowing RL to focus on patch quality rather than format learning.

**Research directions that become less attractive:**
- **Pure distillation from proprietary models as a path to state-of-the-art open models.** If RL on publicly available data can match or exceed distillation-based approaches (41.0% vs. 28.8-32.8% for prior distillation-based open models in Table 1), the strategic case for distillation weakens. Distillation will remain useful when proprietary models are substantially stronger than the best open model (as Claude-3.5-Sonnet at 50.8% still exceeds SWE-RL's 41.0%), but the trajectory suggests that RL on open data may close this gap without requiring ongoing access to proprietary systems that can change or disappear.
- **Execution-based reward at scale for repository-level tasks.** The paper demonstrates that execution is not necessary for effective RL training on real-world patches. While execution-based verification may provide additional signal (particularly for catching functional errors that textually similar patches can introduce), the infrastructure burden of setting up per-repository execution environments is substantial. SWE-RL shows that text-based rewards are sufficient to reach competitive performance, making execution-based approaches a potential refinement rather than a prerequisite.

---

### Follow-Up Research This Work Enables

**Semantic-equivalence-aware reward functions that preserve SWE-RL's gradient properties while reducing false negatives.** The paper's primary acknowledged limitation is that `difflib.SequenceMatcher` rewards textual similarity rather than semantic equivalence—a patch that fixes the bug using different variable names or control flow receives lower reward than a textually-similar but subtly wrong patch. A follow-up could train a lightweight semantic equivalence classifier on pairs of patches from the same PR (where multiple commits represent functionally equivalent refinements of the same fix) and use its output to weight or augment the sequence similarity reward. The specific experiment: fine-tune a small code model (e.g., 7B parameters) on a dataset of patch pairs labeled as "semantically equivalent" or "not equivalent" using the heuristic that different commits within the same merged PR are likely equivalent alternatives, while patches from different PRs are not. Use this classifier's predicted equivalence probability as a multiplicative factor on the sequence similarity reward: `R(o) = compare(patch_pred, patch_gt) × P(equivalent | patch_pred, patch_gt)`. The test would be whether this augmented reward achieves higher repair accuracy than the pure sequence similarity baseline (34.8% in Table 2) while matching or exceeding the continuous reward's training dynamics (Figure 5). A negative result—the classifier adds noise and degrades training—would suggest that the simplicity of sequence similarity is a feature, not a bug, and that the model already learns to avoid textually-similar-but-wrong patches through the optimization process.

**Ablation of SWE-RL across model scales (7B, 13B, 70B, 405B) to characterize the capability threshold for emergent reasoning.** The paper's "aha moments" (Figure 3) and OOD generalization (Table 3) are demonstrated only at 70B scale. A critical open question is whether these emergent properties require a minimum model capacity—below which RL simply teaches surface-level patch formatting without generalized reasoning—or whether they scale smoothly with model size. The experiment: apply the identical SWE-RL pipeline (same seed data, same reward function, same GRPO hyperparameters) to Llama-3.3 variants at 8B, 70B, and (if feasible) 405B parameters. Measure three outcomes at each scale: (a) SWE-bench Verified performance under the standard evaluation protocol, (b) OOD generalization on the five-benchmark suite from Table 3, and (c) quantitative metrics for emergent reasoning behavior, such as the frequency of self-reflection keywords ("wait," "however," "alternatively," "I notice") in thinking blocks, the correlation between thinking block length and patch correctness, and the probability of the model revisiting an earlier assumption in its chain-of-thought. The hypothesis to test: reasoning emergence follows a sigmoid curve where models below ~30B parameters show minimal RL-induced reasoning improvement (only format learning), models in the 30-100B range show partial emergence, and models above ~100B show clear "aha moment" patterns. This would establish a capability threshold that practitioners can use to decide whether SWE-RL is worth the compute investment for their available model scale.

**Comparison of SWE-RL against an SFT model trained on raw PR data without synthetic chain-of-thought, to isolate the RL objective from data format confounds.** The paper's RL-vs-SFT comparison (Tables 2 and 3) confounds the training objective (RL vs. SFT) with the data format (raw patches vs. synthetic demonstrations) and the presence of teacher-generated reasoning traces. A clean ablation would train an SFT model on exactly the same (issue, context, oracle patch) triples used for RL, with the training target being the oracle patch formatted as search/replace edits, and no chain-of-thought supervision—the model must learn to produce patches directly from issue descriptions, without reasoning traces in the training data. The evaluation would compare this "SFT-raw" model against the existing SFT baseline and the RL model on three axes: (a) SWE-bench Verified end-to-end performance, (b) repair-only accuracy with oracle files (Table 2 protocol), and (c) OOD generalization (Table 3 benchmarks). The hypothesis: SFT-raw will outperform the synthetic-data SFT baseline on SWE-bench (because it avoids teacher-generated reasoning errors) but will still degrade on OOD tasks (because SFT's output-matching objective, regardless of data format, steers the model toward the training distribution), while RL will maintain its OOD improvement (because the KL penalty and outcome-based optimization preserve general capabilities). A contradictory result—SFT-raw matches RL's OOD performance—would imply that the generalization benefit is not from RL per se but from avoiding teacher-generated reasoning data, which would redirect research toward better SFT data construction rather than RL algorithms.

**Integration of SWE-RL-trained repair models into agentic scaffolds with execution feedback loops.** The paper's pipeline-based Agentless Mini scaffold (Figure 7) deliberately avoids interactive feedback, but the strongest SWE-bench results come from agentic systems (Claude-3.5-Sonnet with OpenHands at 53.0%, Table 1). A natural extension would test whether the RL-trained model's repair capability transfers to an interactive setting: take Llama3-SWE-RL-70B, deploy it in an agentic scaffold (e.g., OpenHands or SWE-agent) with access to file reading, code search, test execution, and error message observation, and measure whether the model can use execution feedback to iteratively refine patches. The specific experiment: start with the model's single-turn repair (34.8% with oracle files, Table 2) as a baseline, then allow up to 5 rounds of interaction where the model sees test execution results and can revise its patch. Measure (a) how many of the 65.2% of oracle-file cases that fail in single-turn mode become solvable with interaction, (b) whether the model's RL-trained reasoning strategies (self-reflection, alternative exploration) transfer to the interactive setting or whether it requires additional RL training in the interactive environment, and (c) whether the final SWE-bench Verified score approaches the 50%+ range achieved by proprietary agentic systems. A negative result—the model fails to use execution feedback effectively—would suggest that single-turn RL training does not transfer to multi-turn reasoning and that interactive RL training (as in SWE-Gym) is necessary, which would be a significant finding about the limits of static-context RL for complex tasks.

**Difficulty-conditioned allocation of the large generation budget (500 samples) to improve inference efficiency.** The paper's scaling analysis (Figure 4) shows that 500 repair samples are used for all 500 SWE-bench Verified instances, but the marginal benefit of additional samples varies dramatically: some issues are solved with 40 samples, others require 160+, and some are never solved regardless of budget. A follow-up could estimate per-instance difficulty from the first ~20 repair samples (using metrics like average PRM score, agreement rate among top patches, or variance in the predicted patches) and allocate the remaining budget adaptively: easy instances get minimal additional samples, hard instances get the full budget, and instances where the model shows no sign of progress get budget reallocated to more promising instances. The evaluation would measure whether adaptive allocation achieves the same 41.0% solve rate with significantly fewer total samples (e.g., 150 samples average per instance instead of 500), and would characterize the difficulty-dependent scaling curves that the paper's Figure 4 averages over. This would directly address the paper's unacknowledged limitation that the 500-sample protocol is computationally prohibitive for deployment, and would connect SWE-RL to the compute-optimal inference scaling framework described in the reference example paper.

**Cross-lingual and cross-language generalization testing to determine the scope of SWE-RL's reasoning transfer.** The OOD generalization results in Table 3 are compelling but limited to Python-centric, English-language benchmarks. The SWE-RL training data is multi-language (drawn from all of GitHub, 2015-2024) and multi-lingual (issue descriptions in many natural languages), but the evaluation is not. A systematic generalization study would test Llama3-SWE-RL-70B on: (a) SWE-bench Multilingual (Yang et al., 2024c), which extends SWE-bench to non-Python repositories, to test whether the repair capability transfers across programming languages; (b) issue-solving tasks in non-English natural languages (e.g., by translating SWE-bench issue descriptions to Chinese, Japanese, or Russian and evaluating whether the model can still produce correct patches), to test whether the reasoning strategies are language-agnostic; and (c) non-coding reasoning benchmarks that are further from the training distribution than the Table 3 benchmarks, such as legal reasoning (CaseHOLD), scientific reasoning (SciQ), or multi-step planning (StrategyQA), to test the limits of the "generalized reasoning" claim. The hypothesis: SWE-RL's reasoning transfer will be strongest for tasks that share structural properties with software debugging (identifying root causes, tracing dependencies, reasoning about edge cases) regardless of domain, and weakest for tasks requiring factual knowledge or domain-specific heuristics that the RL training did not reinforce.

---

### Practical Applications and Downstream Use Cases

**Automated bug-fixing for open-source repositories with no existing test infrastructure.** Many open-source projects—particularly smaller ones maintained by individual developers—have limited or no automated test suites, making execution-based verification impossible. SWE-RL-trained models could be deployed as automated "first-pass" fixers on these repositories: when a user files a bug report with a clear issue description, the model receives the issue text and the relevant repository files, generates a candidate patch with reasoning, and the patch is submitted as a draft pull request for human review. The 34.8% repair accuracy with oracle files (Table 2, greedy decoding) means that roughly one in three well-localized bugs would receive a correct fix without any human intervention beyond filing the issue. Even a 15-20% success rate on uncurated issues (accounting for localization errors) would represent a meaningful reduction in maintainer burden for projects receiving dozens of bug reports, and the model's reasoning traces (Figure 3) provide transparency that lets maintainers quickly assess whether the proposed fix is credible. The key practical advantage is that this requires no per-project setup—no test environment, no CI integration, no execution infrastructure—just the repository's source code and the issue text.

**Data generation for self-improving code models through iterative RL.** The SWE-RL framework enables a self-improvement loop analogous to the ReST and STaR paradigms but applied to real-world software engineering: (1) train a model on the current seed PR dataset using SWE-RL, (2) use the trained model to generate candidate fixes for a new set of issues (from recent GitHub PRs not in the training set), (3) filter the generated fixes by comparing them to the oracle patches from those PRs (using the same `difflib.SequenceMatcher` reward), keeping fixes above a similarity threshold, (4) add the high-similarity generated fixes to the training dataset, and (5) repeat. The paper's finding that the RL model outperforms the SFT baseline by 5.2 points on repair-only performance (Table 2, 34.8% vs. 29.6%) suggests that each iteration would produce higher-quality training data than the previous iteration, potentially enabling a virtuous cycle of improvement. The 95.6% format accuracy means generated patches are mostly valid, and the continuous reward provides a natural filtering criterion. The practical benefit is that this loop can run autonomously on newly merged PRs without any human annotation, continuously improving the model as more open-source software is developed.

**Cost-efficient fine-tuning for enterprise codebases where proprietary model distillation is prohibited.** Many organizations (particularly in finance, defense, and healthcare) cannot send their proprietary code to external API providers like OpenAI or Anthropic, making distillation from GPT-4o or Claude-3.5-Sonnet legally or contractually impossible. SWE-RL provides a fully on-premises training pipeline: the organization curates its internal PR history (which contains exactly the kind of bug-fix pairs SWE-RL requires), runs the RL training on their own GPU cluster using only open-source models (Llama-3.3-70B-Instruct as the base, `difflib` from the Python standard library as the reward), and deploys the resulting model for internal bug-fixing. The 41.0% SWE-bench Verified result establishes that RL on PR data can produce competitive models without external API calls, and the training cost is quantifiable (512 H100 GPUs for 32 hours). This enables organizations with large internal codebases and strict data residency requirements to build custom software engineering models that would otherwise be impossible to create. The key numbers for a cost-benefit analysis: training costs 512 × 32 = 16,384 GPU-hours on H100s, the resulting model achieves 34.8% single-attempt repair accuracy on well-localized bugs (Table 2), and the alternative (sending code to external APIs for distillation) may be prohibited entirely, making SWE-RL the only viable approach.

**Bootstrapping software engineering capability in low-resource programming languages.** The SWE-RL training data is drawn from all public GitHub repositories regardless of programming language, but the paper only evaluates on Python (SWE-bench Verified). For programming languages with smaller open-source ecosystems and no existing SE benchmarks (e.g., Rust, Go, Kotlin, Swift), SWE-RL offers a way to train bug-fixing models without needing to build language-specific execution environments, test harnesses, or annotated datasets. The recipe: collect merged PRs from GitHub repositories in the target language using the same curation pipeline (Appendix A, Figure 6), format them as (issue, code context, oracle patch) triples, initialize from a multilingual code model (or the base Llama-3.3-70B-Instruct which already has some multi-language capability from pretraining), and run SWE-RL. The sequence similarity reward is fully language-agnostic (it operates on text), requiring no language-specific tooling. The resulting model could be deployed alongside language-specific tooling (compilers, linters, test frameworks) in a scaffold analogous to Agentless Mini, adapted to the target language's build system and testing conventions. The practical impact would be largest for languages where the ratio of open-source code to available developer time is high—newer languages with growing ecosystems but limited tooling—where automated bug-fixing could compound the productivity of the existing developer community.

---

### When to Prefer This Method

The paper articulates a clear tradeoff between SWE-RL and the dominant alternative (SFT on distilled outputs from proprietary models), grounded in specific results from Tables 1-3 and Figure 5:

- **Prefer SWE-RL when:** (a) access to proprietary models (GPT-4o, Claude-3.5-Sonnet) is restricted by cost, API terms, or data privacy requirements—SWE-RL uses only publicly available GitHub data and open-source models; (b) the deployment scenario requires the model to maintain or improve its general capabilities (code generation, math, language understanding) alongside its SE-specific capabilities—Table 3 shows RL preserves or improves OOD performance while SFT degrades it on average; (c) the target domain has abundant historical fix data but limited execution infrastructure—SWE-RL's text-based reward requires no test execution, making it applicable to repositories without test suites; (d) you believe the model can surpass the best available teacher on the target task—RL optimizes for an outcome and can theoretically exceed the performance of any single oracle patch through exploration, while SFT on teacher outputs cannot exceed the teacher's capability on the teacher's own demonstration distribution.

- **Prefer SFT on distilled outputs when:** (a) the strongest proprietary models substantially outperform the best open model on your target task—for SWE-bench Verified, Claude-3.5-Sonnet achieves 50.8% with Agentless (Table 1), suggesting that distilled training data from Claude-3.5-Sonnet may capture capabilities that RL on open data has not yet reached; (b) training compute is extremely limited—SFT on a few thousand high-quality demonstrations may be cheaper than RL requiring 512 H100 GPUs for 32 hours; (c) you need the model to learn specific output formatting or interaction patterns that are easier to demonstrate than to specify as reward functions—the paper's SFT baseline achieved slightly higher format accuracy (96.2% vs. 95.6%, Table 2), suggesting SFT is marginally better at teaching strict output templates; (d) the target task requires multi-turn interactive behavior that single-turn RL training does not teach—the paper's pipeline-based scaffold (Figure 7) deliberately simplifies the interaction model, and training interactive agents may benefit from trajectory-level SFT on human or teacher demonstrations.

These preferences are not mutually exclusive—a hybrid approach (SFT warmup for format learning and basic editing, followed by RL for patch quality optimization) may outperform either alone. The paper does not test such a hybrid, but the complementary strengths (SFT for surface patterns, RL for reasoning quality) suggest it as a natural next step.