Code2025~126 minarXiv:2507.12415

SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

score 10Xinyi He, Qian Liu, Mingzhe Du +5↑ 43★ 48Jul 16, 2025cs.SEhttps://github.com/SWE-Perf/SWE-Perf

🎯 Pitch

Current LLMs achieve barely 2% runtime improvement when autonomously optimizing real-world repositories—versus nearly 11% for human experts. The performance gap widens dramatically as tasks require touching more functions or coordinating changes across modules.

1. Executive Summary

This paper introduces SWE-Perf, the first benchmark designed to evaluate whether large language models can optimize code performance in real-world repository-level contexts rather than isolated function-level exercises. The benchmark comprises 140 carefully curated instances — each derived from performance-improving pull requests across 9 popular GitHub repositories, including the relevant codebase, target functions, performance-related tests, expert-authored patches, and sandboxed Docker environments — and evaluates models under two settings: an oracle file-level setting (where models are given the exact files and functions that the expert modified) and a realistic repo-level setting (where autonomous agents like Agentless and OpenHands must navigate the entire repository). Evaluating 10 leading LLMs including Claude-4-opus, OpenAI-o3, and Gemini-2.5-Pro, the paper reveals a substantial capability gap: even the best-performing agent-based system, OpenHands with Claude-3.7-sonnet, achieves only 2.26% statistically significant performance gain compared to the expert's 10.85%, establishing that current LLMs can rival or surpass experts on narrow individual repositories (e.g., OpenHands outperforms experts by 0.4% on sklearn) but fail to scale their optimization capability to multi-function, high-runtime, and cross-module scenarios where human expertise compounds its advantage.

2. Context and Motivation

The central gap this paper addresses is straightforward to state but profound in its implications: despite explosive progress in LLMs for code generation, bug fixing, and even repository-level software engineering, there exists no benchmark — and consequently no systematic understanding — of whether these models can optimize the runtime performance of real-world codebases. This is not a niche oversight. In production software systems, performance optimization yields system-wide benefits that often exceed the value of incremental feature additions or even bug fixes (Nascimento et al., 2023; Mancebo et al., 2021). Yet the entire LLM-for-code evaluation ecosystem has evolved around functional correctness — "does the code do the right thing?" — while ignoring the equally critical question: "does the code run efficiently?"

This gap is especially striking given the maturity of both the software engineering benchmarking community and the code efficiency evaluation community as separate research threads. On one side, repository-level SWE benchmarks have become increasingly sophisticated and realistic. On the other, code efficiency benchmarks have measured whether models can produce performant implementations. But these two traditions have never intersected, leaving a vacuum at precisely the point where real-world software engineering lives: optimization that spans files, modules, and architecture-level decisions within an existing, large-scale codebase. SWE-Perf positions itself directly in this vacuum, asking the question that neither community has been equipped to answer: given an authentic repository with a demonstrated performance problem, can an LLM produce the cross-cutting, multi-file optimization that solves it?

Why This Problem Matters: Performance Is Not Optional in Production Systems

The paper's motivation is not merely that performance optimization is "nice to have" or academically interesting. It articulates why performance is foundational in ways that correctness-focused benchmarks cannot capture (Section 1):

Performance optimization requires fundamentally different reasoning than bug fixing. A developer fixing a bug has a clear target: locate the flawed logic and correct it. The bug exists as a deviation from intended behavior, and the fix restores the intended behavior. Performance optimization has no such clear target. The code is functionally correct; the question is whether a different, functionally equivalent implementation can achieve the same result faster. This requires reasoning about algorithmic complexity, data structure selection, I/O patterns, caching strategies, memory allocation, and often architectural refactoring — skills that are distinct from, and arguably more sophisticated than, bug-localization-and-fix workflows. The paper notes (Section 2) that efficiency optimization represents an "open-ended problem lacking standardized solutions" that introduces "additional complexities, including identifying optimization targets, designing performance-oriented changes, and requiring long-context understanding, planning capabilities, and efficiency-specific domain knowledge."

The optimization surface area grows exponentially at repository scale. Function-level benchmarks necessarily isolate optimization to a single, self-contained unit of code. In real repositories, optimization opportunities rarely respect function boundaries. A performance bottleneck in one function may be fixable by restructuring a data structure defined in a different module, by changing how a library dependency is invoked, or by introducing caching at a system boundary. The paper emphasizes (Section 1) that "collaborative improvements between files and modules typically unlock far greater optimization potential than isolated function-level changes." This means that function-level benchmarks are not merely simpler — they systematically underestimate the difficulty and overestimate model capability relative to real-world optimization tasks, because they exclude the cross-cutting reasoning that defines expert-level performance work.

The economic and environmental stakes are real. The paper cites literature establishing that software energy efficiency (Pereira et al., 2021) and performance analysis (Mancebo et al., 2021) are critical concerns for large-scale systems. A performance regression in a widely-used library like xarray, scikit-learn, or sympy — all represented in SWE-Perf — ripples across thousands of downstream applications. Conversely, an optimization in such libraries compounds to massive aggregate compute savings. If LLMs are to serve as genuine software engineering assistants, they must be capable not just of writing correct code, but of writing code that does not needlessly waste cycles. A model that produces functionally correct but O(n²)-where-O(n)-would-suffice implementations is not ready for production deployment, regardless of its SWE-Bench score.

Existing benchmarks are not easily repurposed for performance evaluation. The paper makes a subtle but important point about why we cannot simply take SWE-Bench or similar datasets and add performance metrics (Section 1). Performance optimization PRs are rare relative to bug-fix PRs, and identifying them requires running the code — not just analyzing diffs. The absence of a human "optimal" reference implementation makes it unclear whether a given function can be improved further, which is essential for creating a meaningful benchmark. Without expert-authored patches that demonstrably improve performance, evaluators would be left asking whether mediocre model results reflect model limitations or simply a lack of optimization headroom. SWE-Perf solves this by construction: every instance derives from a real PR where an expert developer did achieve measurable improvement, establishing a concrete performance ceiling.

Where Prior Approaches Fall Short: The Bifurcated Evaluation Landscape

The paper identifies two distinct traditions in code LLM evaluation, both of which leave critical gaps that SWE-Perf fills.

Repository-Level SWE Benchmarks: Correctness Over Performance

The dominant paradigm in realistic code LLM evaluation emerged with SWE-Bench (Jimenez et al., 2024), which introduced the concept of evaluating whether LLMs can resolve real GitHub issues by generating patches that pass the repository's existing test suite. This spawned a rich ecosystem of related benchmarks: SWE-Gym (Pan et al., 2024) scaled the data for training software engineering agents, SWE-Dev (Du et al., 2025) focused on feature-driven development, SWE-Lancer (Miserendino et al., 2025a) introduced economic valuation of freelance tasks, and SWT-Bench (Mündler et al., 2024) emphasized test-based validation of bug fixes.

These benchmarks share a critical limitation: they evaluate only functional correctness, typically measured as whether the model's patch causes the repository's test suite to pass. This is a natural and important metric, but it is orthogonal to performance. A model could produce a patch that passes all tests but introduces an O(n³) algorithm where the original used O(n log n), and the benchmark would score it as a success. Conversely, a model could produce a patch that dramatically improves performance but fails one edge-case test, and the benchmark would score it as a failure. The evaluation framework is simply not designed to capture performance as a dimension of quality.

More specifically, the paper notes that these benchmarks "are primarily tailored for tasks with well-defined objectives, such as bug fixing" (Section 2). Bug fixing has a clear correctness criterion: the bug is gone when the previously-failing tests now pass. Performance optimization has no equivalent binary signal. A 5% improvement might be significant or noise depending on measurement methodology and variance. The absence of standardized performance evaluation infrastructure — statistically rigorous runtime measurement, warm-up procedures, outlier filtering, significance testing — means that even if researchers wanted to evaluate performance on these benchmarks, they would lack the methodological scaffolding to do so reliably. SWE-Perf provides this infrastructure as a first-class design element (Phase 4 of data collection, Section 3.2).

Code Efficiency Benchmarks: Isolated Functions Without Repository Context

Parallel to the SWE benchmarking tradition, several datasets have targeted code efficiency specifically: Mercury (Du et al., 2024), EFFIBENCH (Huang et al., 2024), EvalPerf (Liu et al., 2024), and KernelBench (Ouyang et al., 2025). These benchmarks ask models to generate efficient implementations of algorithmic problems (e.g., "write a function to compute the nth Fibonacci number efficiently") and evaluate the runtime of the generated code.

The paper identifies two fundamental limitations that prevent these benchmarks from answering the repository-level performance question (Section 2):

They constrain optimization to function boundaries. In these benchmarks, the model writes a single function from scratch. There is no existing codebase to navigate, no cross-module dependencies to understand, no architectural decisions to reconsider. The model's entire task is contained within the function signature. This is a dramatic simplification relative to real-world optimization, where the most impactful changes often involve rethinking how components interact rather than micro-optimizing a single function's implementation. As the paper puts it, these benchmarks "overlook the complexity of real-world efficiency challenges that span multiple files and modules" and "limit their ability to benchmark models' capabilities in addressing cross-cutting concerns such as dataflow refactoring or parallelism, where optimization potential is typically more substantial."

They lack the authentic optimization targets that real repositories provide. In Mercury or EFFIBENCH, the model is asked to produce any correct and efficient implementation. There is no expert reference showing what level of optimization is achievable, and no pre-existing codebase that constrains the solution space. In real repositories, optimizers must work within the constraints of existing APIs, compatibility requirements, and codebase conventions. The expert patch in SWE-Perf demonstrates not just that improvement is possible, but how an expert navigated these constraints to achieve it, providing a grounded performance ceiling that isolated benchmarks cannot offer.

A Concurrent Effort: GSO

The paper acknowledges one concurrent work, GSO (Shetty et al., 2025), which also identifies performance-improving commits. The key distinction is methodological: GSO "identifies performance-improving commits by combining an LLM-based judge with code-change heuristics," while SWE-Perf "leverages pull requests and runtime environments for identification" (Section 2). In other words, GSO uses an LLM to judge whether a commit likely improved performance, whereas SWE-Perf directly measures runtime before and after in controlled environments. The measurement-based approach provides ground-truth performance data rather than proxy judgments, which is essential for establishing a reliable benchmark where performance improvement is a verified fact rather than an estimated likelihood.

LLM-Based SWE Methods: Not Designed for Open-Ended Optimization

The paper also evaluates representative methods from the SWE-bench ecosystem — Agentless (pipeline-based) and OpenHands (agent-based) — and makes the important observation that these methods "were not originally designed for open-ended code performance optimization, leaving room for adaptation and further exploration" (Section 2). These systems incorporate components like fault localization, patch generation, and regression testing that are optimized for bug-fixing workflows. Performance optimization introduces different requirements: rather than localizing a fault, the system must identify optimization opportunities (which may be spread across multiple locations, none of which are "buggy"); rather than ensuring tests still pass, the system must verify that performance actually improved; and rather than producing a minimal fix, the system may need to make architecturally significant changes. The paper's use of these methods as baselines is not a claim that they are optimized for the task, but rather an assessment of how well current SOTA repository-level SWE approaches transfer to the performance domain — an assessment that, as the results show, leaves substantial room for improvement.

How SWE-Perf Positions Itself

The paper's positioning can be understood as filling the intersection of two established research directions — repository-level SWE evaluation and code efficiency measurement — by building infrastructure that neither direction has provided. The key design decisions that realize this positioning are:

Ground-truth performance improvement as a prerequisite for inclusion. Unlike existing benchmarks that include any PR regardless of its performance characteristics, SWE-Perf requires every instance to exhibit a statistically significant, verified performance improvement between the original and patched codebases (Phase 4, Algorithm 1). This ensures that (1) there is real optimization headroom, (2) the expert patch provides a concrete performance ceiling, and (3) model-generated patches can be meaningfully compared against a known achievable improvement. The paper explicitly uses the expert patch "not only to confirm the feasibility of improvement but also as a human-derived gold standard against which to evaluate LLM-generated code performance optimization edits" (Section 3.1).

Statistically rigorous performance measurement as a first-class evaluation primitive. The paper invests heavily in measurement methodology that is absent from prior SWE benchmarks: Docker-based environment standardization with constrained CPU cores (1 core during collection, 5 during evaluation) and memory (16 GB), warm-up runs to mitigate initialization effects, 20-repetition runtime measurements with IQR-based outlier filtering, and Mann-Whitney U tests with a conservative δ-computation algorithm (Algorithm 1) to establish statistically significant minimum performance gains. This methodology acknowledges that runtime measurement is inherently noisy and that naive before/after comparisons can produce spurious "improvements" from environmental variation rather than genuine code changes.

Two difficulty tiers that disentangle optimization from retrieval. The Oracle (file-level) and Realistic (repo-level) settings serve distinct evaluation purposes. The Oracle setting tests whether models can generate performance-improving code when given the exact target functions and files — essentially measuring pure optimization capability without navigation or retrieval overhead. The Realistic setting tests whether autonomous systems can navigate a repository, identify what needs changing, and then execute those changes — measuring end-to-end capability including retrieval, planning, and multi-step reasoning. This decomposition allows the paper to diagnose where models fail: is the bottleneck code generation itself, or is it the repository-scale reasoning required to identify what to generate?

Coverage across diverse, widely-used repositories. The 140 instances span 9 repositories ranging from numerical computing (xarray, 54 instances) and machine learning (scikit-learn, 32 instances) to symbolic mathematics (sympy, 20 instances) and scientific computing (astropy, 12 instances), ensuring that findings are not an artifact of a single codebase's characteristics. The repositories are drawn from the same set as SWE-Bench, establishing continuity with the broader SWE evaluation ecosystem while pivoting the evaluation target from correctness to performance.

In essence, the paper argues that the field has developed sophisticated tools for evaluating whether LLMs can make code correct, and separate tools for evaluating whether LLMs can write efficient algorithms, but has no tool for evaluating whether LLMs can make existing, large-scale codebases faster — which is what practicing software engineers actually spend a substantial fraction of their time doing. SWE-Perf is that tool, and its initial results suggest that current LLMs are far from ready for this task.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

What the system is: SWE-Perf is not a single software system but rather a benchmark and evaluation framework — a dataset of 140 real-world code performance optimization tasks, each packaged with the original codebase, expert-authored performance-improving patches, executable Docker environments, and statistically rigorous performance measurement infrastructure.

What problem it solves and the "shape" of the solution: The fundamental problem is that no existing benchmark can answer whether LLMs can optimize code performance in authentic, repository-scale software engineering contexts. The solution is shaped as a data collection pipeline that mines GitHub pull requests for verified, statistically significant performance improvements, and an evaluation protocol that measures LLM-generated patches against human expert baselines using three progressively stringent metrics (Apply, Correctness, Performance) under two difficulty settings (Oracle file-level and Realistic repo-level).

3.2 Big-picture architecture (diagram in words)

The SWE-Perf framework has five major components, arranged in a pipeline that flows from raw GitHub data to benchmark instances, and then from model-generated patches to evaluation scores:

Data Collection Pipeline (Phase 1–5, Section 3.2): A five-stage filtering process that starts with ~102,241 pull requests across 12 popular repositories, isolates those with measurable performance improvements, verifies those improvements are statistically stable, and extracts optimization targets for both oracle and realistic evaluation settings. This component produces the 140 benchmark instances.
Benchmark Instance Structure (end of Section 3.2): Each of the 140 instances is a self-contained package comprising six elements: (a) the original codebase source, (b) a sandboxed Docker environment, (c) target functions for optimization (both oracle and realistic versions), (d) performance-related unit tests, (e) pre-computed runtime metrics for both original and expert-patched codebases, and (f) the expert-authored patch serving as the gold-standard reference.
Evaluation Protocol (Section 4): A three-tier metric hierarchy — Apply (does the patch apply cleanly?), Correctness (do all performance-related tests still pass?), and Performance (what is the statistically significant minimum performance gain?) — that evaluates model outputs against the expert baseline using identical measurement methodology as the data collection phase.
Model Baselines (Section 5.1): Representative methods from three paradigms: direct chain-of-thought prompting (Oracle setting with 10 LLMs), pipeline-based approaches (Agentless with Claude-3.7-sonnet), and agent-based systems (OpenHands with Claude-3.7-sonnet). These baselines span the current state-of-the-art in repository-level software engineering.
Analysis Infrastructure (Section 5.3): Tools for dissecting model performance along four dimensions: decoupling performance from correctness, analyzing the impact of target function count, examining runtime-dependent optimization capability, and comparing modification strategies via word cloud analysis of added lines.

Information flows as follows: raw GitHub PRs enter the collection pipeline → filtering and statistical verification produces 140 benchmark instances → models receive instances under Oracle or Realistic settings → models produce patches → patches are applied and evaluated against the three-tier metric hierarchy → performance scores are analyzed across difficulty dimensions.

3.3 Roadmap for the deep dive

First, the data collection pipeline (Phases 1–5): This is the foundation — understanding how instances are constructed explains what makes SWE-Perf's evaluation meaningful. We'll walk through each phase's filtering criteria, thresholds, and design rationale, since the benchmark's quality depends entirely on these choices.
Second, the statistically significant minimum performance gain (Algorithm 1): This is the paper's key methodological innovation for measurement rigor. We'll unpack the algorithm's pessimistic adjustment mechanism and why it's necessary for reliable benchmarking.
Third, the target function extraction strategy (Phase 5 detail): This explains the Oracle vs. Realistic distinction and why the paper uses dynamic execution tracing rather than naive approaches.
Fourth, the three-tier evaluation framework (Section 4): With the benchmark constructed, we'll examine how model outputs are scored — the Apply → Correctness → Performance progression and how each metric is computed.
Fifth, the baseline configurations (Section 5.1): The specific models, prompts, and hyperparameters used in evaluation, since these determine what "state-of-the-art" means in context.
Sixth, the analysis methodology (Section 5.3): How the paper dissects failure modes — decoupling correctness from performance, examining scaling behavior with function count and runtime, and keyword-based strategy comparison.

3.4 Detailed, sentence-based technical breakdown

This is primarily a benchmark construction and empirical evaluation paper whose core idea is that evaluating LLM code performance optimization requires (1) a dataset where every instance has a verified, statistically significant performance improvement ceiling, and (2) a measurement methodology that can reliably distinguish genuine optimization from environmental noise.

Phase 1: Collect Pull Requests with Performance Potential

The data collection begins by following established methodologies from SWE-Bench (Jimenez et al., 2024) and SWE-Gym (Pan et al., 2024), with one critical modification to the filtering criteria that enables the focus on performance rather than correctness.

Step 1: Repository selection. The paper adopts the same 12 repositories used in SWE-Bench. These are high-star, popular GitHub projects selected for their real-world relevance and active development communities. The choice ensures continuity with the broader SWE evaluation ecosystem while pivoting the evaluation target.

Step 2: PR crawling. The authors re-crawl these repositories rather than reusing existing SWE-Bench PRs because their subsequent filtering criteria differ — specifically, they do not require that PRs contribute tests, which excludes many SWE-Bench instances.

Step 3: Attribute filtering. This is where SWE-Perf diverges from prior work. SWE-Bench and SWE-Gym use two main filtering criteria: (1) the PR resolves an issue, and (2) the PR contributes tests. SWE-Perf retains only criterion (1) and explicitly drops criterion (2). The reasoning is important: criterion (2) exists in SWE-Bench to ensure that the PR can be evaluated via the repository's test suite, which is essential for correctness-based evaluation. But for performance evaluation, the paper cares about whether the PR changes execution time — not whether it adds new tests. Requiring contributed tests would filter out performance-only PRs where the developer improved efficiency without modifying the test suite, which is actually the common case for performance work (the tests already exist and should continue to pass). By relaxing this constraint, SWE-Perf casts a wider net specifically for performance-relevant PRs.

At the conclusion of Phase 1, 102,241 pull requests have been collected, from which 19,797 remain after filtering. The massive attrition (roughly 81% removed) occurs because many PRs do not resolve issues or do not meet the structural criteria for evaluation.

Phase 2: Measure CodeBase Performance at Scale

This phase converts the filtered PRs into performance measurements by building and testing each codebase's original and modified versions in controlled environments. It is, by the paper's own characterization, "the most time-consuming step in the entire data collection pipeline" (Section 3.2, Phase 2, Step 2).

Step 1: Build Docker environments. For each PR, the original and modified codebases must be built into executable Docker containers. Following SWE-Gym's approach, the paper constructs a Docker image for each codebase version, with codebases that fail to build being excluded. The Docker-based approach serves two purposes: it ensures reproducibility by capturing the exact dependency environment, and it enables resource constraint enforcement — each container is limited to a single CPU core and 16 GB of memory during this phase. This CPU constraint is critical because unrestricted parallelism would make runtime measurements incomparable across containers that might use different numbers of cores. By forcing single-core execution, the paper ensures that performance differences reflect algorithmic improvements rather than parallelization differences (which would be an orthogonal optimization dimension).

Step 2: Execute unit tests. Using pytest, the paper runs all unit tests inside each Docker container. The scale here is substantial: by this stage, 34,397 distinct codebases have been gathered (original and modified versions across all PRs), and test execution succeeds on 19,499 of them. The paper provides a striking example of the computational demands: in the xarray repository, each codebase contains on average over 220,000 test cases, and testing a single codebase can take over one hour on a single-core CPU. Table 4 in Appendix B provides detailed runtime statistics across repositories, showing that sklearn codebases average 83.89 minutes for test execution (max 119.96 minutes), sympy averages 24.60 minutes (max 112.17 minutes), and xarray averages 58.11 minutes (max 119.95 minutes). The total execution time for this phase across all repositories is substantial — likely thousands of CPU-hours — which explains why the paper uses only two Linux machines, each with 256 logical CPU cores and 2.0 TiB RAM (Appendix B).

Step 3: Record runtimes. pytest collects the execution time for each individual unit test. To address measurement noise, each codebase is evaluated in three repeated experimental runs, producing three runtime measurements per unit test. These triplicate measurements feed into the statistical filtering in Phase 3. Codebases for which runtimes cannot be successfully collected are excluded. The three-fold repetition is a deliberate design choice: it provides enough data to compute means and assess variance without making the already-expensive collection pipeline prohibitively slower. It is worth noting that Phase 4 later uses 20 repetitions for final verification — the 3-repetition design in Phase 2 is a screening-level compromise between statistical rigor and computational feasibility.

Design rationale for environment standardization: The paper takes two specific steps to minimize environmental effects: (1) limiting each container to one CPU core and 16 GB memory, and (2) running three repeated measurements. The single-core constraint is the more important design choice. In real-world optimization, parallelization is a legitimate performance strategy, but benchmarking it requires controlling for hardware parallelism, which varies across machines. By restricting to single-core, the paper isolates algorithmic efficiency from parallelization strategy, making measurements comparable and ensuring that improvements reflect genuine code quality rather than "we threw more cores at it." This is a methodological narrowing that the paper acknowledges implicitly — future work could extend SWE-Perf to multi-core optimization, but the initial benchmark establishes a clean single-core baseline.

Phase 3: Identify Performance-Optimizing Pull Requests

With per-test runtime data for all surviving codebases, Phase 3 filters for PRs where the code modifications demonstrably and substantially improve performance.

Step 1: Filter PRs with performance optimization. For each PR, the paper has three runtime measurements per unit test in both the original and modified codebases. The filtering uses two criteria:

(1) Correctness criterion: The unit test must pass in both the original and modified codebases. This is captured as the pytest result being "pass" in both cases. This criterion is essential because a performance measurement on a failing test is meaningless — the test might terminate early on an assertion failure, producing a misleadingly short runtime. It also ensures that performance improvements are not achieved by breaking functionality, which would violate the correctness-preserving requirement of real-world optimization.

(2) Performance Ratio criterion: The paper computes an optimized ratio that quantifies the relative speedup:

$\text{Ratio} = \frac{R_{\text{original}} - R_{\text{modified}}}{R_{\text{original}}}$

where $R_{\text{original}}$ is the mean runtime of the unit test across the three replicates on the original codebase, and $R_{\text{modified}}$ is the corresponding mean on the modified codebase.

What it computes: For each unit test, the ratio captures the fraction of original runtime eliminated by the modification. A ratio of 0.3 means the modified version runs 30% faster than the original (i.e., the modified runtime is 70% of the original). A ratio of 0.0 means no improvement (or degradation). The ratio is signed such that positive values indicate improvement.

Why this form: The ratio normalizes by the original runtime, making it comparable across tests with vastly different absolute runtimes (a 0.1-second improvement on a 0.3-second test is proportionally larger than a 1-second improvement on a 10-second test). This is the standard speedup metric in performance engineering. The paper requires the ratio to be below a specified threshold of 0.3, meaning the optimization must achieve at least a 30% relative speedup. This is a deliberately aggressive threshold — it excludes marginal improvements (e.g., 5–10% speedups) that might be measurement noise or not practically meaningful. The choice of 0.3 represents a judgment about what constitutes a "substantial" optimization worth including in a benchmark.

An important subtlety: the paper states the threshold as "below 0.3" but the ratio formula produces values between 0 and 1 for improvements (where 0 means no change and 1 would mean the modified version takes zero time — impossible in practice). "Below 0.3" would actually mean less improvement, which seems inverted. The paper likely means that the remaining runtime fraction (i.e., $R_{\text{modified}} / R_{\text{original}}$ ) must be below 0.3, or equivalently that the ratio must be above 0.7. The exact interpretation matters for understanding the selectivity of the filter, but the paper's wording is slightly ambiguous. What is unambiguous is the outcome: only PRs with very substantial performance improvements survive this filter.

Step 2: Select unit tests that execute human patches. After the ratio-based filtering, the paper performs dynamic execution analysis to ensure performance improvements are genuinely attributable to the code modifications in the PR. Using dynamic execution tracing (the paper does not specify the exact tool, but the methodology involves running the tests and checking code coverage of patched functions), the paper identifies unit tests that satisfy two conditions: (1) they exercise the patched code segments modified in the PR — meaning the test's execution path actually passes through the changed functions, establishing a causal link between the patch and any performance change; and (2) they do not execute any unit tests that were themselves modified within the PR — this prevents contamination where a test change (rather than a source code change) drives the performance difference.

This step serves a crucial verification purpose. Without it, a unit test might show a performance improvement in the modified codebase that is actually due to a change in the test itself (e.g., the expert simplified the test), or due to an unrelated code change that happened to be in the same PR but is not the optimization target. By requiring that the test exercise patched code without being patched itself, the paper establishes a clean causal attribution.

At the conclusion of Phase 3, from the 4,413 PRs that had runtime data available for both original and modified codebases, 1,696 valid instances are derived. The attrition from 19,499 to 4,413 reflects the reality that many codebases have dangling or incompatible main and development branches. The further reduction to 1,696 reflects the stringency of the ratio and dynamic execution filters — genuine, substantial, and causally attributable performance improvements are relatively rare in open-source PRs.

Phase 4: Verify Stable Performance Improvements Through Statistical Testing

The 1,696 instances from Phase 3 represent PRs with promising performance characteristics, but three measurements per test is insufficient to establish that the improvement is stable (not an artifact of measurement noise) and statistically significant (unlikely to occur by chance given runtime variance). Phase 4 applies rigorous statistical methodology to filter for only those instances where the performance gain is reliable.

Step 1: Add warm-up. Before each performance measurement, the paper executes three performance-related unit tests to warm up the environment. Warm-up is essential in performance benchmarking because initial executions incur one-time costs — JIT compilation, CPU cache warming, disk I/O caching, Python module imports — that inflate runtime and introduce variance. By running warm-up tests before measurement, the paper ensures that these startup effects do not contaminate the timing data. The choice of three warm-up runs is a common heuristic; it provides enough iterations for most startup effects to stabilize without excessive overhead.

Step 2: Execute 20 repetitions. Each unit test is run 20 times, substantially more than the 3 repetitions in Phase 2. This increase reflects the transition from screening (where approximate measurements suffice) to verification (where statistical tests require sufficient sample sizes). With 20 samples per test, the paper can compute reliable distributional statistics (quartiles for outlier detection, rank-based tests for significance) that would be underpowered with only 3 samples.

Step 3: Filter outliers. Runtime measurements are notoriously noisy due to operating system scheduling, garbage collection pauses, and other transient effects. The paper uses the Interquartile Range (IQR) method with a threshold multiplier $k = 1$ . The filtering criteria are:

$\text{Outlier if } r_i < Q_1 - k \times \text{IQR} \quad \text{or} \quad r_i > Q_3 + k \times \text{IQR}$

where $Q_1$ is the first quartile (25th percentile) of the 20 runtime measurements, $Q_3$ is the third quartile (75th percentile), and $\text{IQR} = Q_3 - Q_1$ is the interquartile range.

What it computes: For each unit test, the IQR method defines a "normal range" extending $k$ IQR widths below $Q_1$ and above $Q_3$ . Any measurement falling outside this range is classified as an outlier and removed from the sample. With $k=1$ , the bounds are relatively tight — approximately 2.7 standard deviations for normally distributed data — meaning moderately extreme values are excluded.

Why this form: The IQR method is robust to the very outliers it detects, because quartiles are rank-based statistics that are minimally affected by extreme values. Mean-and-standard-deviation-based outlier detection would be inappropriate here because the outliers themselves would inflate the standard deviation, masking the very effect the method aims to detect. The choice of $k=1$ is conservative — standard practice often uses $k=1.5$ , which produces wider bounds (the standard "boxplot" definition). The tighter $k=1$ means the paper is aggressive about removing potential noise, prioritizing measurement purity over sample retention. This is a defensible choice for a benchmark where false positives (calling noise a "real improvement") are more damaging than false negatives (excluding borderline cases).

Step 4: Calculate statistically significant performance gain (Algorithm 1). This is the paper's key methodological contribution to performance measurement rigor. The goal is to compute, for each unit test, a conservative minimum performance gain called $\delta$ (delta) — the largest performance improvement that can be attributed to the modification with statistical confidence, after pessimistically weakening the observed improvement.

Here is the algorithm in full detail:

Input: Two arrays of filtered runtimes — $A = [a_1, a_2, ..., a_n]$ from the original codebase and $B = [b_1, b_2, ..., b_m]$ from the modified codebase. Also: a significance level $\alpha = 0.1$ , a gain increment step of $0.01$ , and a maximum gain to test of $1.0$ .

Output: $\delta$ , the conservative minimum significant performance gain (a value between 0 and 1).

Procedure:

Initialize $x = 0.0$ and $\delta = 0.0$ . Here $x$ represents the candidate gain being tested, and $\delta$ stores the largest gain that has passed the significance test.
While $x \leq \text{max\_x}$ (1.0):
- Create an adjusted modified array $B_{\text{adj}} = B \times (1 - x)$ . This pessimistically weakens the observed improvement: each modified runtime is scaled toward the original by factor $(1-x)$ . When $x = 0$ , there is no adjustment (the observed improvement is used as-is). As $x$ increases, the modified runtimes are artificially inflated, making the improvement appear smaller. When $x$ equals the true fractional improvement, $B_{\text{adj}}$ should be statistically indistinguishable from $A$ .
- Perform a one-sided Mann-Whitney U test with $B_{\text{adj}}$ and $A$ , with alternative hypothesis "greater" — meaning we test whether $B_{\text{adj}}$ is stochastically larger than $A$ (i.e., whether the adjusted modified runtimes are actually slower than the original, indicating we've weakened the improvement too much). The test produces a p-value.
- If $p < \alpha$ (0.1), then $B_{\text{adj}}$ is still statistically significantly larger than $A$ even after adjustment. This means the adjustment $x$ was insufficient to eliminate the observed improvement, so we can claim at least $x$ as a significant gain. Update $\delta = x$ , then increment $x = x + \text{step}$ (0.01) and continue testing larger gains.
- If $p \geq \alpha$ , then the adjustment has eliminated or reversed the statistical significance of the improvement — $B_{\text{adj}}$ is no longer significantly faster than $A$ . Stop the search and return $\delta$ .
Return $\delta$ .

What it computes: $\delta$ is the largest fraction of speedup that remains statistically significant even after conservatively adjusting the observed improvement. For example, if the observed mean speedup is 40% but $\delta = 0.25$ , the algorithm is saying: "we are statistically confident that the true improvement is at least 25%, even though the point estimate is 40%." The gap between the observed improvement and $\delta$ represents measurement uncertainty absorbed by the conservative adjustment.

Why this form: This algorithm addresses a fundamental challenge in performance benchmarking: runtime measurements are distributions, not point values, and the observed improvement may be inflated by lucky scheduling, cache effects, or other transient factors. A naive approach of comparing means would claim an improvement any time $\text{mean}(B) < \text{mean}(A)$ , even if the difference is within the noise floor. A standard significance test (e.g., t-test comparing means) would establish that there is some nonzero improvement but would not quantify the minimum improvement we can confidently claim. The pessimistic adjustment approach solves this by systematically weakening the observed improvement until the statistical signal disappears, producing a conservative lower bound on the true effect size.

The choice of the Mann-Whitney U test (rather than a t-test) is deliberate and important. The Mann-Whitney test is non-parametric — it does not assume normality of the runtime distributions, which is appropriate because runtime distributions are often skewed (bounded below by zero, with long right tails due to rare slow executions). The one-sided "greater" alternative tests specifically whether the adjusted modified runtimes are larger than the original runtimes — equivalent to testing whether the improvement has been over-adjusted away. Using $\alpha = 0.1$ is relatively lenient (standard practice often uses 0.05), which means the algorithm is slightly biased toward claiming a nonzero $\delta$ . The increment step of 0.01 provides 1% granularity in the reported gain.

Final filter: Unit tests with $\delta$ exceeding a threshold of 0.05 (5% minimum significant improvement) are retained. This final threshold ensures that even the lower bound of the improvement is practically meaningful. Only 140 instances survive this entire pipeline — from the original 102,241 PRs, representing a 0.14% yield. This extreme filtering demonstrates how rare verified, stable, substantial performance improvements are in open-source repositories, and underscores the value of a curated benchmark over ad-hoc performance evaluation.

Phase 5: Extract Optimization Targets for Two Evaluation Settings

With the 140 verified performance-improving instances identified, the final phase extracts the specific functions that models should target for optimization. This phase bifurcates into two settings that serve different evaluation purposes.

Setting 1: Oracle (File-Level). This setting provides models with the exact functions that the expert developer modified, along with the entire files containing those functions. The target functions are extracted by combining AST (Abstract Syntax Tree) analysis with unified diff matching on the human patch. AST analysis identifies the syntactic boundaries of modified functions; diff matching identifies which functions the patch actually touched. The combination ensures that only genuinely modified functions are included — not functions that happen to be in the same file but were untouched by the expert's optimization.

The Oracle setting evaluates pure code generation capability. By telling the model exactly what to optimize, it removes the retrieval and localization challenges that dominate real-world optimization workflows. If models fail in the Oracle setting, the bottleneck is in generating performance-improving code itself — not in finding where to apply it. If models succeed in Oracle but fail in Realistic, the bottleneck is in repository-scale navigation and identification.

Setting 2: Realistic (Repo-Level). This setting provides models with the functions that were measured during performance testing — the functions directly invoked by the performance-related unit tests — rather than the functions the expert actually modified. These are not necessarily the same functions: the expert might have optimized a helper function deep in the call stack that is not directly called by the test but whose performance affects the test's runtime. The Realistic setting therefore requires the model to trace the call graph, identify bottlenecks, and determine which functions need modification.

The identification process for Realistic targets: The paper uses yappi (Yet Another Python Profiler) to record all functions dynamically executed during the performance-related unit tests. Combined with AST parsing of the unit test code, the paper determines the specific functions directly invoked by each test. Crucially, the paper explicitly avoids using the unit test itself as a target, preventing what it calls "test information leakage" — a scenario where the model, seeing the test as a target function, could "optimize" by pruning the target function to retain only the specific functionality exercised by the test, achieving a speedup at the cost of breaking untested functionality. By excluding the test from the target set and requiring all existing tests to continue passing (enforced by the Correctness metric), the benchmark guards against this degenerate optimization strategy.

The Realistic setting evaluates end-to-end autonomous optimization capability. The model (or agent system) must navigate the repository, understand the codebase, identify optimization opportunities, and implement changes — all without being told which functions the expert modified. This is substantially harder and more representative of how human developers approach performance work.

Design rationale for two settings: The paper uses this bifurcation to disentangle code generation from repository navigation. By comparing Oracle and Realistic performance, future researchers can diagnose whether a system's weakness lies in generating optimizations (both settings would show poor performance) or in finding where to apply them (Oracle would show good performance while Realistic shows poor performance). This decomposition is essential for targeted improvement — there is no point building better retrieval mechanisms if the model cannot generate good patches even when pointed at the right code.

The resulting benchmark instance: After all five phases, each of the 140 SWE-Perf instances comprises six components:

CodeBase: The source code of the original (pre-patch) repository version.
Executable Environment: A Docker image and container configured to execute the original codebase with constrained resources.
Target Functions: Two lists — the Oracle target functions (directly modified by the expert) and the Realistic target functions (measured during testing).
Performance-related Unit Tests: The specific tests identified in Phase 3 as exercising patched code and showing stable performance improvements.
Runtime Metrics: The original and modified codebase runtime measurements for all performance-related tests, including the 20-repetition data and computed $\delta$ values.
Expert Patch: The human-authored diff that achieved the verified performance improvement, serving as the gold-standard reference.

Instance statistics (Table 1): The 140 instances span 9 repositories (the remaining 3 from the original 12 produced zero qualifying instances after filtering). The average codebase has 447.3 non-test files and 170,000 non-test lines of code. Expert patches average 131.1 lines edited across 4.3 files and 7.6 functions, demonstrating that real performance optimization often requires cross-cutting changes rather than single-line tweaks. The performance-related tests average 8.1 per instance, with original runtimes averaging 0.28 seconds (max 25.2 seconds). The performance ratio averages 10.9% but can reach 87.8%, indicating that some optimizations nearly eliminate the runtime of targeted operations.

The Three-Tier Evaluation Framework (Section 4)

With the benchmark constructed, the evaluation protocol applies a three-metric hierarchy to assess model-generated patches:

Metric 1: Apply. This is a binary gate: can the model-generated patch be applied to the original codebase without conflicts or errors? The metric is computed as:

$\text{Apply} = \frac{N_{\text{apply}}}{N_{\text{total}}}$

where $N_{\text{apply}}$ is the number of instances where the patch applies cleanly and $N_{\text{total}}$ is the total number of instances (140).

What it computes: The fraction of model-generated patches that are syntactically valid and can be mechanically applied to the codebase. A failure here means the model produced a malformed patch — wrong file paths, incorrect line numbers, or syntax errors in the diff format.

Why this exists: Apply is a surprisingly discriminating metric. Models that produce high-quality code may still fail at the mechanical task of producing a correctly formatted unified diff. This metric captures a basic engineering competency that is orthogonal to optimization skill but essential for practical deployment.

Metric 2: Correctness. For successfully applied patches, this metric assesses functional preservation:

$\text{Correctness} = \frac{\sum_{i=1}^{N_{\text{total}}} \left[ \bigwedge_{j=1}^{N_i} \text{result\_post}_{i,j} = \text{pass} \right]}{N_{\text{total}}}$

where $\text{result\_post}_{i,j}$ is the pytest result for the $j$ -th performance-related unit test on the $i$ -th instance after the patch is applied, $N_i$ is the number of performance-related tests for that instance, and $\bigwedge$ is logical AND — the instance counts as correct only if every single performance-related test passes.

What it computes: The fraction of instances where the model's optimization preserves all existing functionality. An instance with even one failing test is counted as incorrect, regardless of how many other tests pass. This is a strict criterion that prevents models from "optimizing" by breaking functionality.

Why this form: The all-or-nothing correctness criterion mirrors real-world deployment requirements. A performance optimization that breaks any existing test would not be accepted in practice, regardless of the speedup achieved. By using logical AND across all tests, the metric penalizes partial breakage severely — aligning with the engineering reality that correctness is non-negotiable.

Metric 3: Performance. For instances that are both applied and correct, this metric computes the statistically significant minimum performance gain using the same methodology as Phase 4 of data collection:

$\text{Performance} = \frac{1}{N_{\text{total}}} \sum_{i=1}^{N_{\text{total}}} P_i, \quad P_i = \frac{1}{n_i} \sum_{j=1}^{N_i} p_{i,j}$

where $p_{i,j}$ is the minimum performance gain (computed via Algorithm 1) for test $j$ of instance $i$ , $N_i$ is the number of tests for that instance, and $n_i$ is an instance-level normalization factor.

What it computes: The average, across all 140 instances, of the per-instance average minimum significant performance gain. For instances where the patch failed to apply or broke correctness, $P_i$ is implicitly zero (since no valid performance measurement is possible). This means Performance is a composite metric that penalizes both application failures, correctness regressions, and weak optimization — a single number that captures the end-to-end utility of the model's output.

Why this form: The Performance metric design reflects an important prioritization: a 5% speedup that preserves correctness is better than a 50% speedup that breaks one test. By implicitly setting $P_i = 0$ for incorrect patches, the metric encodes the engineering value hierarchy where correctness dominates performance. A model could not game this metric by producing aggressive but incorrect optimizations — those would score zero on Correctness and thus zero on Performance.

Measurement standardization for evaluation: The paper re-evaluates the original codebase runtime during the testing phase even when the original runtime was already known from data collection. This re-measurement ensures full comparability — both the original and post-patch runtimes are collected in the same environment, at the same time, with the same Docker container state, eliminating any systematic drift in measurement conditions that could inflate or deflate apparent improvements.

Evaluation environment for LLM evaluation: During evaluation, Docker containers are constrained to 5 CPU cores (increased from 1 during data collection) and the same 16 GB memory. The increase to 5 cores reflects a practical compromise: the evaluation needs to be computationally feasible across 140 instances and multiple model baselines, and single-core constraint would make the evaluation pipeline prohibitively slow without adding measurement value (since the original-to-modified comparison is done within the same core count).

Baseline Configurations (Section 5.1)

The paper evaluates three categories of methods:

Oracle (Direct Model Prompting). Ten LLMs are evaluated under a chain-of-thought prompting strategy where the model receives the Oracle target functions and the entire files containing them. The prompt (shown in Figure 14, Appendix C) follows a structured format:

Problem statement: Instructions to "enhance the computational efficiency and execution speed across the entire repository" with clarification that optimization may be achieved "either directly through modifications to the objective functions or indirectly by improving computationally intensive subroutines."
Code context: The content of the relevant files.
Output format: The model must produce SEARCH/REPLACE blocks using a specific format with file paths, search blocks, dividing lines, and replace blocks.

The prompt includes four specific conditions that constrain the optimization: (1) acceleration of at least one objective function is sufficient, (2) optimization may be direct or indirect through subroutines, (3) maximal efficiency gains should be prioritized where feasible, and (4) all existing unit tests must remain unaltered to preserve functional correctness.

Model-specific configurations:

OpenAI/GPT: Versions o1-preview-2024-09-12, o3-2025-04-16, and gpt-4o-2024-11-20, with temperature 0.2, top-p 0.1, and maximum 8,192 tokens.
Claude: Versions gcp-claude37-sonnet, gcp-claude4-opus, and gcp-claude4-sonnet, with thinking feature enabled (2,000 token thinking budget) and maximum 8,192 token output.
DeepSeek: Versions deepseek-r1-0528 and DeepSeek-V3.
Gemini: Version gemini-2.5-pro-preview-05-06.
Qwen: Version Qwen3-235B-A22B.

The temperature and top-p settings (0.2 and 0.1 respectively) are relatively low, favoring deterministic, conservative generation over exploration — appropriate for a task where correctness preservation is critical and creative optimization must be balanced against the risk of breaking functionality.

Agentless (Pipeline-Based). Agentless (Xia et al., 2024) follows a fixed multi-stage workflow: hierarchical fault localization to identify relevant code regions, code repair to generate patches, and candidate patch selection through regression and reproduction testing. The paper sets the sample number to 1. Agentless was designed for bug-fixing tasks, so its localization and repair stages are oriented toward correcting faulty logic rather than identifying optimization opportunities. The evaluation therefore tests how well a bug-fixing pipeline transfers to performance optimization without modification.

OpenHands (Agent-Based). OpenHands (Wang et al., 2024) provides a flexible platform for autonomous software development agents that can iteratively reason, execute commands, and modify code across multiple steps. The paper configures it with a maximum of 50 iterations and uses Claude-3.7-sonnet as the base model — the officially recommended backend for OpenHands, reported to work best within the framework. The 50-iteration limit constrains how many reasoning-and-action cycles the agent can perform per instance, providing a realistic budget for autonomous exploration.

Both Agentless and OpenHands use Claude-3.7-sonnet as their base model, enabling a controlled comparison where the difference in performance can be attributed to the system architecture (pipeline vs. agent) rather than the underlying model capability.

Analysis Methodology (Section 5.3)

Beyond raw benchmarking, the paper conducts four diagnostic analyses to understand where and why models fall short of expert performance:

1. Performance Decoupled from Correctness (Section 5.3.1). The standard Performance metric conflates optimization skill with patch-generation correctness — a model that produces brilliant optimizations that fail to apply cleanly scores zero. To isolate pure optimization capability, the paper recomputes Performance using only correct examples as the denominator:

$\text{Performance\_pass} = \frac{1}{N_{\text{correctness}}} \sum_{i \in \text{correct}} P_i$

This metric answers: "For the subset of instances where the model managed to produce a correct patch, how good was the optimization?" The expert reference is also recomputed on the same subset, ensuring fair comparison.

2. Impact of Target Function Count (Section 5.3.2). The paper bins instances by the number of target functions (both Oracle and Realistic) and computes average performance within each bin. This analysis tests the hypothesis that optimization difficulty scales with the number of functions requiring modification — a core claim about why repository-level optimization is harder than function-level benchmarks.

3. Runtime-Dependent Optimization Capability (Section 5.3.3). Instances are binned by original codebase runtime and performance is computed per bin. This tests whether models are better at optimizing short-running functions (where micro-optimizations like reducing constant factors matter) or long-running functions (where algorithmic improvements are needed). The expert performance trend serves as a baseline for the achievable improvement at each runtime scale.

4. Keyword Analysis of Modification Strategies (Section 5.3.4). The paper generates word clouds from the lines added in model-generated patches (OpenHands) and expert patches, comparing the vocabulary of modifications. This qualitative analysis reveals what kinds of changes each approach favors: low-level infrastructure modifications vs. high-level algorithmic restructuring. The word clouds in Figures 10, 11, 12, and 13 (main text and Appendix C) provide visual evidence for claims about differing optimization strategies.

4. Key Insights and Innovations

Innovation 1: Performance Optimization Is a Distinct Reasoning Paradigm, Not Just "Bug Fixing Without the Bug"

The paper's most fundamental conceptual move is establishing that repository-level performance optimization constitutes a qualitatively different task from correctness-oriented software engineering, not merely a variant of the same underlying capability. This reframing matters because the entire LLM-for-code evaluation ecosystem — from SWE-Bench to SWE-Lancer — has implicitly treated software engineering tasks as lying along a single difficulty spectrum, where more complex bug fixing approximates the challenges of other software engineering activities. SWE-Perf argues, through its construction and results, that this assumption is wrong.

The distinction turns on what the paper calls the "open-ended" nature of performance optimization (Section 2). Bug fixing has a clear correctness criterion: the bug is resolved when a previously-failing test now passes. The optimization target is well-defined, and the model's task is to localize a fault and produce a minimal fix. Performance optimization has no equivalent binary signal. The code already works correctly — the question is whether a functionally equivalent but faster implementation exists, which may require rethinking data structures, algorithms, caching strategies, or architectural relationships across files and modules. The optimization target is not a "buggy line" to fix but rather a design choice to reconsider, and the solution space is unbounded (there is always potentially a faster implementation).

This conceptual distinction has concrete implications that the paper's results bear out. The significant performance gap between even the best models and expert performance — OpenHands achieves only 2.26% gain vs. the expert's 10.85% (Table 2) — cannot be explained by deficiencies in code generation or localization alone, since models already demonstrate these capabilities on SWE-Bench. Something else is missing. The paper's keyword analysis (Figures 10–11, Section 5.3.4) provides suggestive evidence for what that "something else" is: model-generated patches focus on "low-level data structures and basic functionality" (terms like "children," "identifier," "attributes"), while expert patches emphasize "high-level abstractions and data integrity" (terms like "literal," "value," "type," "dtype"). The models are optimizing at the wrong level of abstraction — they tweak implementation details rather than reconsidering design choices, which is where substantial performance gains typically originate.

Comparison to prior assumptions: Prior to SWE-Perf, the dominant assumption — embodied in the design of SWE-Bench, Agentless, and OpenHands — was that repository-level software engineering is primarily a challenge of localization and patch generation. If a model can find where to change code and produce a correct change, it has solved the core problem. SWE-Perf demonstrates that performance optimization introduces an additional cognitive demand: efficiency reasoning, the ability to analyze a correct implementation and identify which correct implementation among many would be fastest. This capability is not tested by any existing benchmark and appears to be weakly developed in current LLMs.

Significance beyond the benchmark: This reframing suggests that future progress on repository-level performance optimization will require more than incremental improvements to existing SWE methods. It will require fundamentally new capabilities — perhaps specialized training on performance-aware code transformations, explicit algorithmic complexity reasoning, or architectures that can simulate the performance characteristics of alternative implementations. The paper's contribution is not just a dataset but a problem definition that reveals a capability gap invisible to existing evaluations.

The paper's bifurcation into Oracle (file-level) and Realistic (repo-level) settings is not merely an experimental convenience — it is a diagnostic framework that decomposes the monolithic "optimize this repository" task into two separable capabilities: (1) the ability to generate performance-improving code when told exactly what to modify, and (2) the ability to navigate a repository and identify optimization opportunities autonomously.

This decomposition is significant because prior work in SWE evaluation — both benchmarks like SWE-Bench and systems like Agentless and OpenHands — treats the end-to-end task as the evaluation target, making it impossible to diagnose where a system fails. A low SWE-Bench score could reflect weak fault localization, poor patch generation, or inadequate regression testing — but the benchmark provides no mechanism to attribute failure to specific components. SWE-Perf's two-setting design enables precisely this attribution for performance optimization.

The paper's Oracle results provide evidence for why this decomposition matters empirically. In the Oracle setting, where models receive the exact target functions and files, the best models still achieve only 1.76% performance gain (Claude-4-sonnet) compared to 10.85% for the expert (Table 2). This means that even when the localization problem is completely removed — when the model is handed the answer to "where should I optimize?" — current LLMs cannot generate optimizations approaching human quality. The bottleneck is not (primarily) retrieval; it is optimization reasoning itself. Conversely, the gap between Oracle and Realistic for the same base model (Claude-3.7-sonnet: 1.24% Oracle vs. 2.26% Realistic via OpenHands, Table 2) is modest and actually reversed — the Realistic setting with an agent architecture slightly outperforms Oracle with direct prompting — suggesting that agent-based iteration can partially compensate for weaker initial generation, or that the Oracle prompt format is suboptimal relative to the agent's multi-step refinement capability.

Comparison to prior work: Agentless and OpenHands were both designed to improve end-to-end SWE-Bench scores. Their architectures — hierarchical localization, repair, regression testing — are optimized for the fault-localization-and-fix paradigm. SWE-Perf's Oracle/Realistic decomposition reveals that these architectures transfer imperfectly to performance optimization: Agentless achieves only 0.41% performance gain (Table 2), worse than even the weakest Oracle models, suggesting that its pipeline components actively harm performance optimization (perhaps by filtering out "large" changes that look like regressions but are actually beneficial refactorings). This is a diagnostic finding that would be invisible in an undifferentiated end-to-end benchmark.

Significance as a methodological contribution: The Oracle/Realistic decomposition is a reusable evaluation design pattern that future benchmarks — whether for performance optimization, security hardening, accessibility improvement, or any other non-correctness software quality dimension — can adopt. By establishing two difficulty tiers that isolate generation from navigation, SWE-Perf provides a template for creating benchmarks that not only measure capability but explain it.

Innovation 3: Statistically Rigorous Performance Measurement as a First-Class Benchmark Primitive

The paper's most operationally innovative contribution is the measurement methodology embodied in Algorithm 1 and the surrounding infrastructure — warm-up runs, 20-repetition measurement, IQR-based outlier filtering, and the pessimistic δ-computation using Mann-Whitney U tests. This methodology represents a fundamental departure from how code performance has been evaluated in prior LLM benchmarks.

What the field did before: Existing code efficiency benchmarks — Mercury, EFFIBENCH, EvalPerf, KernelBench — evaluate model-generated code by running it and comparing runtime to reference implementations. But their measurement methodology is typically opaque: they may use single measurements, lack warm-up, ignore runtime variance, and report point estimates without confidence bounds. This is adequate for ranking models on isolated algorithmic problems where performance differences are often large (orders of magnitude between naive and optimal implementations), but it is inadequate for benchmarking incremental optimizations on real codebases where improvements may be 20–40% — large enough to matter, but small enough that measurement noise and environmental variation can produce spurious "improvements" that are not reproducible.

SWE-Perf's methodology directly addresses this measurement reliability problem. The pessimistic δ-computation (Algorithm 1) is particularly clever: rather than reporting the observed improvement — which could be inflated by lucky scheduling or transient system states — it systematically weakens the observed improvement until the statistical signal disappears, producing a conservative lower bound on the true effect. This means that when SWE-Perf reports that a model achieved a 1.24% performance gain (Claude-3.7-sonnet in Oracle, Table 2), the reader can be confident that this is a genuine, reproducible improvement — not an artifact of measurement methodology.

Why this matters beyond SWE-Perf: The measurement infrastructure is not merely a detail of this specific benchmark; it is a contribution to evaluation methodology for the entire field of LLM code optimization. Future benchmarks that evaluate code performance — whether at the function level or repository level — can adopt this methodology to ensure that reported improvements are reliable. The paper's choice of the Mann-Whitney U test (non-parametric, robust to skewed distributions) and the pessimistic adjustment algorithm provide a concrete recipe that other researchers can replicate.

Significance as infrastructure: This is arguably the paper's most lasting contribution. Datasets age (new models will saturate SWE-Perf's 140 instances, requiring expansion), but measurement methodology that establishes statistical rigor for performance benchmarking addresses a permanent challenge in the field. The paper's careful documentation of measurement parameters — warm-up runs, repetition counts, outlier detection thresholds, significance levels, and the δ-computation algorithm — provides a reference implementation that raises the bar for all future code performance evaluation.

Innovation 4: The "Performance Ceiling" Concept — Expert Patches as Upper Bounds, Not Oracles

A subtle but important conceptual move in SWE-Perf's design is the treatment of expert patches not as ground-truth "correct" answers (as in bug-fixing benchmarks) but as evidence of a feasible performance ceiling. The paper explicitly acknowledges (Appendix A.1) that "human-written patches... may not represent the optimal achievable performance, potentially underestimating the true upper bound of improvement." This framing distinguishes performance optimization from correctness-oriented tasks in a way that has implications for benchmark interpretation.

In bug-fixing benchmarks, the expert patch is a ground-truth solution — there is a specific bug, and the patch resolves it. A model that produces a different but equally correct fix is still scored as successful. The expert patch serves as a verification mechanism (this bug is fixable) rather than a target.

In SWE-Perf, the expert patch serves a different role. It demonstrates that at least a certain level of improvement is achievable. The expert's 10.85% average performance gain is not the "right answer" — it is a lower bound on what is possible, and it is possible (even likely) that a better optimizer could exceed it. This has two implications for how benchmark results should be interpreted:

A model that matches the expert's performance is not "solved" — it has reached a known feasible ceiling, but there may be further headroom. The paper's finding that OpenHands outperforms the expert by 0.4% on sklearn (Figure 4) is evidence that the expert patches are not optimal, and that models can discover optimizations the human developer missed.
A model that underperforms the expert is not necessarily "bad at optimization" — it may be optimizing the wrong targets, or its optimizations may be incompatible with the evaluation constraints (e.g., breaking correctness). The Performance metric's composite nature (implicitly zeroing incorrect patches) means that a model could be producing genuinely faster code that fails one edge-case test, and it would score zero — not because its optimization is poor, but because its correctness preservation is insufficient.

Comparison to prior benchmarks: SWE-Bench and its derivatives treat the expert patch as the definition of correctness: if the model's patch makes all tests pass, it succeeds. SWE-Perf's more nuanced treatment of the expert patch — as a feasibility demonstration rather than a correctness criterion — acknowledges that performance optimization is inherently open-ended and that the expert's solution is one point in a large space of possible optimizations, not the answer. This is a more philosophically honest characterization of the optimization task.

Significance for future research: The performance ceiling concept suggests that SWE-Perf's most interesting future results may not be models approaching 100% of the expert baseline, but rather models that significantly exceed the expert baseline on specific instances — demonstrating that automated optimization can discover improvements that human experts missed. The paper's early evidence of this (OpenHands on sklearn) hints at this potential. As models improve, the expert patches transition from "targets to reach" to "baselines to surpass," and SWE-Perf's design accommodates this transition in a way that correctness-only benchmarks cannot.

Innovation 5: Difficulty Characterization Along Multiple Orthogonal Dimensions Reveals Where Models Fail

The paper's analysis framework (Section 5.3) introduces a multi-dimensional difficulty characterization that goes beyond the single "pass/fail" metric common in SWE benchmarks. By analyzing model performance as a function of target function count, original runtime magnitude, and repository identity, the paper constructs a rich profile of where current LLMs succeed and fail — and, by implication, what capabilities they lack.

The key findings from this analysis are individually subtle but collectively paint a coherent picture of model limitations:

Target function count matters. Figure 8 shows that expert performance declines as the number of Realistic target functions increases — indicating that multi-function optimization is genuinely harder — but that OpenHands degrades faster than the expert at higher function counts. This suggests that models struggle with the compositional reasoning required to coordinate changes across multiple functions, while human experts can maintain a coherent optimization strategy across broader scope.

Runtime magnitude matters. Figure 9 reveals that expert performance improves with longer original runtimes (more headroom for optimization in slow functions), while model performance plateaus. This implies that models are failing to identify the large-scale algorithmic improvements that make long-running functions faster; they may be applying the same micro-optimization strategies (constant-factor tweaks) regardless of runtime scale, which helps for fast functions but cannot address the fundamental bottlenecks in slow ones.

Repository identity matters profoundly. Figure 4 shows extreme variance in model performance across repositories: OpenHands achieves ~14.5% gain on sklearn but near-zero on sympy (0.7%) and matplotlib (0.2%). Expert performance shows similar variation (from ~31.8% on xarray to ~2.6% on seaborn), but the model-expert gap is not uniform — it is narrow on some repositories and catastrophic on others. This suggests that repository-specific characteristics (codebase complexity, domain knowledge requirements, optimization surface area) strongly modulate model capability, and that aggregate benchmark scores obscure this heterogeneity.

Comparison to prior work: SWE-Bench reports per-repository breakdowns but does not systematically analyze performance along structural dimensions like function count or runtime magnitude. The paper's multi-dimensional analysis is a methodological contribution that could be retrofitted to existing benchmarks to provide richer diagnostic information about model failure modes.

Significance as a diagnostic framework: The difficulty characterization transforms SWE-Perf from a leaderboard (which models score highest?) into a diagnostic instrument (what specific capabilities does a given model lack?). A model that performs well on single-function, short-runtime instances but poorly on multi-function, long-runtime instances likely has adequate code generation but weak compositional reasoning and algorithmic analysis. A model that performs well on sklearn but poorly on sympy may have strong numerical computing knowledge but weak symbolic mathematics knowledge. This granularity enables targeted research: rather than "improving performance on SWE-Perf," researchers can aim to "improve multi-function optimization capability" or "improve symbolic mathematics optimization," with clear evaluation signals for whether they have succeeded.

The paper's word cloud analysis (Section 5.3.4) complements the quantitative dimensions with a qualitative one: what kinds of optimizations do models attempt? The finding that model patches emphasize "low-level data structures" while expert patches emphasize "high-level abstractions" provides a concrete hypothesis about the nature of the capability gap — models are micro-optimizers, not architectural thinkers — that could guide future training data curation, prompting strategies, or system design.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses all 140 instances of the SWE-Perf benchmark, drawn from 9 popular GitHub repositories (xarray: 54 instances, scikit-learn: 32, sympy: 20, astropy: 12, sphinx: 8, seaborn: 6, pylint: 3, matplotlib: 3, requests: 2). Each instance includes the original codebase, performance-related unit tests, expert-authored patches, and sandboxed Docker environments. There is no train/validation/test split — all 140 instances are used for evaluation, since this is a benchmark, not a training corpus.
Base model(s). The paper evaluates 10 LLMs under the Oracle setting and 2 systems under the Realistic setting. Oracle models include: Claude-3.7-sonnet, Claude-4-sonnet, Claude-4-opus, GPT-4o, OpenAI-o1, OpenAI-o3, DeepSeek-V3, DeepSeek-R1, Gemini-2.5-Pro, and Qwen3-235B-A22B. For Realistic evaluation, both Agentless and OpenHands use Claude-3.7-sonnet as the base model. Model choices span frontier proprietary systems (Claude, GPT, Gemini), open-weight models (DeepSeek, Qwen), and both standard and reasoning-specialized variants (o1, o3, R1), providing broad coverage of the current capability landscape. The paper states Claude-3.7-sonnet is used for Agentless and OpenHands because it is "the officially recommended backend for OpenHands" and "has been reported to work best within OpenHands" (Section 5.1).
Metrics. Three hierarchical metrics are computed for every model on every instance:
- Apply: Binary — does the model-generated patch apply cleanly to the original codebase without conflicts or errors? Computed as N_apply / N_total where N_total = 140.
- Correctness: Binary per instance — do all performance-related unit tests pass after the patch is applied? An instance counts as correct only if every single test passes (logical AND across tests). Computed as the fraction of 140 instances meeting this criterion.
- Performance: Continuous — the statistically significant minimum performance gain, averaged across all 140 instances. For each instance that is both applied and correct, the per-test minimum gain p_{i,j} is computed via Algorithm 1 (Mann-Whitney U test with pessimistic adjustment, α=0.1, step=0.01). The instance-level performance is P_i = (1/n_i) * Σ p_{i,j}, and the overall Performance metric is the average of P_i across all 140 instances, with P_i implicitly treated as zero for instances where the patch failed to apply or broke correctness. This composite design means Performance penalizes both application failures, correctness regressions, and weak optimization in a single number. The expert baseline achieves 10.85% Performance (Table 2), representing the average statistically significant minimum gain achieved by human-authored patches, computed identically.
Baselines. Three categories of methods are evaluated:
- Oracle (Direct Model Prompting): Ten LLMs evaluated under chain-of-thought prompting where the model receives the Oracle target functions (the exact functions the expert modified) and the complete files containing them. A single-pass inference generates the patch. This baseline measures pure code generation capability with localization removed.
- Agentless (Xia et al., 2024): A pipeline-based approach using a fixed multi-stage workflow of hierarchical fault localization, code repair, and candidate patch selection through regression and reproduction testing. Sample number is set to 1.
- OpenHands (Wang et al., 2024): An agent-based system providing autonomous software development agents with iterative reasoning and multi-step interaction. Maximum iterations set to 50. Both Agentless and OpenHands use Claude-3.7-sonnet as the base model.
The expert performance (10.85% in Table 2) serves as the human reference, computed by applying expert-authored patches and measuring their statistically significant minimum performance gain using the identical protocol.
Generation budget / compute accounting. For Oracle models, compute is measured implicitly by the single-pass inference — one prompt, one generation per instance. For Agentless and OpenHands, the compute budget is controlled by architecture: Agentless uses 1 sample in its pipeline, and OpenHands uses a maximum of 50 iterations (reasoning-and-action cycles) per instance. The paper does not report token counts, inference FLOPs, or wall-clock time for model runs, making cross-method compute comparisons qualitative rather than quantitative. Docker containers during evaluation are constrained to 5 CPU cores and 16 GB memory (increased from the single-core constraint during data collection to make evaluation computationally feasible across all instances and baselines).
Cross-validation / statistical protocol. No cross-validation is used — this is a benchmark evaluation, not model training. The statistical protocol for performance measurement mirrors Phase 4 of data collection: warm-up (3 runs of performance-related tests), 20 repeated measurements per test, IQR-based outlier filtering with k=1, and δ-computation via Algorithm 1 (Mann-Whitney U test, α=0.1, step=0.01, max gain tested = 1.0). The original codebase runtime is re-measured during evaluation alongside the post-patch runtime, even when original runtime data exists from collection, to ensure full environmental comparability.

Main Quantitative Results

Aggregate Benchmark Performance (Table 2)

Table 2 presents the complete results across all methods and models. The central finding is that no model or system approaches expert-level performance, with the best system (OpenHands) achieving only 2.26% Performance compared to the expert's 10.85% — a gap of 8.59 percentage points.

Oracle (File-Level) Results:

The Apply rates reveal substantial variance in basic patch-generation competency:

Gemini-2.5-Pro achieves the highest Apply rate at 95.00%, followed by Claude-4-opus at 85.71% and OpenAI-o3 at 78.57%.
DeepSeek-V3 (47.85%) and Qwen3-235B-A22B (54.29%) show that even frontier open-weight models struggle with the mechanical task of producing correctly formatted SEARCH/REPLACE blocks — roughly half of their generated patches cannot be applied at all.
The Apply-to-Correctness drop-off varies by model: Gemini-2.5-Pro loses 11.43 percentage points from Apply to Correctness (95.00% → 83.57%), while Claude-4-opus loses 7.14 points (85.71% → 78.57%), suggesting that some models produce patches that apply cleanly but break functionality.

Performance scores are uniformly low. The best Oracle performance comes from Claude-4-sonnet at 1.76%, followed by Gemini-2.5-Pro at 1.48% and OpenAI-o3 at 1.37%. Even the highest-scoring Oracle model achieves only about one-sixth of the expert's 10.85% performance gain. The gap between models is relatively compressed: all Oracle models fall within the range of 0.41% (OpenAI-o1) to 1.76% (Claude-4-sonnet), a spread of only 1.35 percentage points. This compression is striking because it suggests that current LLM capability for performance optimization saturates quickly — moving from a mid-tier model (GPT-4o at 0.60%) to a top-tier model (Claude-4-sonnet at 1.76%) yields only about a 1-percentage-point improvement, while the gap to expert performance remains roughly 9 percentage points. Bigger or more recent models are not qualitatively changing the optimization picture; they are making marginal improvements within a fundamentally capability-limited regime.

Realistic (Repo-Level) Results:

The three Realistic methods show divergent performance:

OpenHands achieves 87.86% Apply, 77.86% Correctness, and 2.26% Performance — the highest Performance of any method or model in the entire evaluation.
Agentless achieves 88.57% Apply (highest overall), but only 70.71% Correctness and 0.41% Performance — the lowest Performance among Realistic methods and worse than most Oracle models.
For comparison, the same base model (Claude-3.7-sonnet) in the Oracle setting achieves 66.43% Apply, 61.43% Correctness, and 1.24% Performance.

The OpenHands vs. Agentless comparison is revealing: Agentless has a slightly higher Apply rate (+0.71 percentage points) but substantially lower Performance (-1.85 percentage points). This suggests that Agentless's pipeline — designed for bug fixing — actively harms performance optimization. Its hierarchical fault localization may identify "suspicious" code regions that are actually correct but suboptimal, and its candidate patch selection (via regression testing) may filter out patches that make substantial changes, since large changes are more likely to introduce test failures in a correctness-oriented pipeline. OpenHands's agent-based approach, with its ability to iteratively reason and refine, achieves better optimization at the cost of slightly lower patch applicability.

The fact that OpenHands (2.26%) outperforms Oracle Claude-3.7-sonnet (1.24%) despite the Oracle setting providing perfect localization information is counterintuitive and important. It implies that single-pass generation, even with perfect context, is insufficient for performance optimization — the iterative refinement capability of the agent architecture provides benefits that outweigh the information advantage of the Oracle setting. This finding challenges the assumption that better localization is the primary bottleneck for performance optimization and suggests instead that multi-step reasoning and refinement are essential.

Per-Repository Performance Breakdown (Figure 4)

Figure 4 disaggregates Performance by repository, comparing Oracle (Claude-3.7-sonnet), Agentless, OpenHands, and Expert. The starkest finding is the extreme heterogeneity across repositories:

xarray (54 instances): Expert achieves 31.8% Performance — the highest of any repository. OpenHands reaches 5.1%, Agentless 2.7%, Oracle 0.5%. The expert-model gap of 26.7 percentage points on xarray is the largest absolute gap in the figure, indicating that xarray optimizations require capabilities (likely domain-specific numerical computing knowledge and cross-module dataflow restructuring) that models fundamentally lack despite xarray being the most represented repository.
sklearn (32 instances): Expert achieves 14.5% Performance. OpenHands achieves 14.5% — matching or slightly exceeding the expert, which the paper characterizes as "an early-stage breakthrough" (Section 5.2). Agentless reaches 3.2%, Oracle 7.4%. This is the only repository where any method reaches expert parity, and it is the strongest evidence that models can optimize code performance under the right conditions.
sympy (20 instances): Expert achieves 3.6%, OpenHands 0.7%, Agentless 1.2%, Oracle 0.4%. The expert performance on sympy is relatively low (3.6%), yet models still underperform dramatically. The paper's word cloud analysis (Figure 11) shows that expert patches for such repositories emphasize domain-specific terms like "sympy" and "time," suggesting that symbolic mathematics optimization requires specialized knowledge that general-purpose models lack.
astropy (12 instances): Expert achieves 4.4%, OpenHands 2.7%, Agentless 0.8%, Oracle 0.8%. The relatively narrow expert-OpenHands gap (1.7 points) on astropy contrasts with the wide gap on xarray, suggesting astropy optimizations may be more similar to the micro-optimization patterns that models can handle.
Others (seaborn: 6, sphinx: 8, pylint: 3, matplotlib: 3, requests: 2): Performance is universally low for both models and experts, with most values below 1%. The low expert performance on these repositories indicates limited optimization headroom — the expert patches achieved relatively small improvements, leaving models with little to match.

The takeaway from Figure 4 is that aggregate Performance scores (Table 2) obscure massive repository-level variance. OpenHands's 2.26% aggregate score is an average of 14.5% on sklearn, ~5.1% on xarray, and near-zero on most other repositories. The benchmark is effectively dominated by performance on xarray and sklearn, and conclusions about model capability must be conditioned on repository characteristics.

Performance Decoupled from Correctness (Figures 5 and 6, Section 5.3.1)

To isolate optimization capability from patch-generation correctness, the paper recomputes Performance using only instances where the model's patch was both applied and correct. This Performance_pass metric answers: for the subset of instances where the model successfully preserved functionality, how close did it come to expert-level optimization?

Figure 5 compares Oracle (Claude-3.7-sonnet), Agentless, and OpenHands against the expert, with the expert score recomputed on each method's correct subset:

OpenHands achieves ~11.4% Performance_pass compared to the expert's ~11.4% (on OpenHands-correct instances). The expert performance is notably higher on this subset than the aggregate 10.85% (Table 2), suggesting that instances where OpenHands produced correct patches tend to have higher optimization potential. The near-parity between OpenHands and expert on this subset indicates that when OpenHands successfully preserves correctness, its optimizations are comparable in magnitude to the expert's — a much more optimistic picture than the aggregate 2.26% suggests.
Oracle achieves 2.0% Performance_pass vs. the expert's 8.3% on Oracle-correct instances — showing that even on instances where Oracle patches are correct, the optimization quality is substantially below expert.
Agentless achieves 0.6% Performance_pass vs. the expert's 8.8% — the worst optimization quality even on correct patches.

Figure 6 extends this analysis across all Oracle models:

OpenAI-o3 achieves 12.2% Performance_pass — slightly above the expert's 11.6% on o3-correct instances. This is notable: on the subset where o3 preserves correctness, it can potentially outperform the expert.
Claude-4-sonnet achieves 10.0% Performance_pass (expert: 8.5%), Claude-4-opus achieves 8.3% (expert: ~8.3%), and Gemini-2.5-Pro achieves 9.7% (expert: ~8.9%).
DeepSeek-V3 and DeepSeek-R1 achieve only 1.3% and 1.8% respectively, substantially below their expert references (~6.5% and ~8.9%), indicating that even on correct patches, these models generate weak optimizations.

The Performance_pass analysis reveals a crucial dynamic: the aggregate Performance gap is driven primarily by correctness failures, not optimization weakness, for the strongest models. Models like OpenAI-o3 and Claude-4-sonnet are capable of expert-level optimization when they manage to produce correct patches. The bottleneck is that they rarely do: OpenAI-o3's Correctness rate is 76.43% (Table 2), meaning roughly a quarter of its patches break functionality, and this rate is high relative to other models. The optimization reasoning capability exists but is not sufficiently coupled with functional correctness preservation — suggesting that future progress may come from better verification mechanisms (e.g., more sophisticated testing, static analysis) rather than fundamentally better optimization strategies.

Impact of Target Function Count on Performance (Figures 7 and 8, Section 5.3.2)

The paper bins instances by the number of target functions and examines how performance varies as a function of this count.

Oracle target functions (Figure 7):

Expert performance remains relatively stable as Oracle function count increases, hovering between roughly 8% and 15% across most bins. There is no clear declining trend, suggesting that when the expert knows exactly which functions to modify, the number of targets does not substantially affect achievable improvement.
Oracle (Claude-3.7-sonnet) performance is uniformly low (below 3%) across all function counts, with no clear pattern. The model's performance is already near-zero for single-function instances, so adding more functions cannot make it meaningfully worse. This is consistent with the interpretation that the Oracle setting's bottleneck is generation quality, not target count.

Realistic target functions (Figure 8):

Expert performance shows a clear declining trend: from roughly 20% at 1 function to roughly 8% at 2 functions, then declining further to near-zero at 32+ functions. This confirms the paper's claim that multi-function optimization is genuinely harder — more target functions mean more code to understand, more potential interactions to consider, and a more constrained optimization space.
OpenHands performance is close to the expert at low function counts (1–2 functions) but diverges sharply at higher counts. At 1–2 functions, OpenHands achieves roughly 10–15%, comparable to the expert's ~15–20%. At 4–8 functions, OpenHands falls to ~5% while the expert remains at ~10%. At 16+ functions, OpenHands drops to near-zero while the expert settles at ~2–5%.
Agentless performance is near-zero across all function counts, consistent with its aggregate result.

The widening gap at higher function counts is perhaps the most diagnostically informative result in the paper. It demonstrates that models can handle simple, localized optimizations (1–2 target functions) at near-expert levels, but their capability degrades rapidly as the optimization scope expands. This supports the paper's core argument that repository-level optimization is qualitatively different from function-level optimization — not just harder by degree, but requiring capabilities (cross-module reasoning, compositional strategy formation, global constraint satisfaction) that current models lack. The expert's relative resilience to increasing function count (declining but not collapsing) suggests that human optimizers can maintain a coherent optimization strategy across broader scope, while models lose coherence as the problem scales.

Impact of Runtime on Performance (Figure 9, Section 5.3.3)

Figure 9 bins instances by the original codebase runtime (in seconds) and plots Performance for Expert, Oracle, Agentless, and OpenHands.

Expert performance increases with runtime: from near-zero at very short runtimes (<0.01 s) to roughly 15–20% at runtimes above 1.28 seconds. This makes intuitive sense: longer-running functions have more optimization headroom (there is more "waste" to eliminate), and the expert can identify and exploit this headroom. The progressive upward trend suggests experts are effective at both micro-optimizations (constant-factor improvements in short functions) and macro-optimizations (algorithmic improvements in long functions), with the latter yielding larger percentage gains.
Model performance plateaus or declines at longer runtimes: OpenHands achieves roughly 5–8% at the shortest runtimes (<0.01–0.02 s) but drops to near-zero at runtimes above 0.32 seconds. Oracle shows a similar pattern, with minor fluctuations but no sustained improvement at longer runtimes. Agentless is near-zero throughout.

This pattern is the inverse of what optimization capability would predict: the expert achieves its largest gains on the instances with the most headroom, while models achieve their (modest) gains primarily on short-running functions and essentially zero on long-running ones. The paper interprets this as evidence that models are applying the same optimization strategies regardless of runtime magnitude — likely micro-optimizations like constant-factor tweaks, import caching, or minor refactoring — and these strategies help for fast functions (where such tweaks are a meaningful fraction of total runtime) but are negligible for slow functions (where the bottleneck is algorithmic or architectural, requiring changes the models cannot identify or implement).

This finding has sharp implications for the practical utility of LLM-driven performance optimization. Long-running functions are precisely where optimization matters most in production systems — a 20% improvement on a function that takes 0.01 seconds saves microseconds per call, while a 20% improvement on a function that takes 10 seconds saves 2 seconds per call. The fact that models fail precisely on the high-impact instances means that even if aggregate Performance scores were higher, the optimization would be misdirected toward low-impact targets. Future work on model-based optimization must specifically target the ability to reason about algorithmic complexity and identify architectural bottlenecks in long-running code paths.

Keyword Analysis of Modification Strategies (Figures 10–13, Section 5.3.4)

The paper generates word clouds from lines added in patches to compare what kinds of modifications models and experts make.

OpenHands patches (Figure 10): Dominated by terms like "children," "identifier," "time," "attributes," "importlib," "frozen," "miniconda3," "envs," and encoded fragments like "bootstrapu0000." The paper interprets this as evidence that OpenHands focuses on "low-level data structures and basic functionality" — structural components, attribute handling, environment configuration, and dependency management. The presence of encoded fragments ("u0000") suggests automated generation targeting syntactic adjustments or toolchain compatibility rather than semantic optimization.

Expert patches (Figure 11): Dominated by terms like "literal," "value," "type," "label," "dtype," "workspace," "sympy," "time," "active packages." The paper interprets this as focusing on "high-level abstractions and data integrity" — type safety, type annotations, data values, and domain-specific computational workflows (symbolic mathematics, resource management, runtime efficiency).

Oracle patches (Figure 13, Appendix C): The Oracle (Claude-3.7) word cloud shows a mix of terms including "identifier," "children," "type," "value," "literal," and "dtype" — intermediate between the OpenHands and Expert distributions, with some high-level abstraction terms appearing but lower-level structural terms still prominent.

Agentless patches (Figure 12, Appendix C): Similar low-level focus to OpenHands, with terms like "identifier," "children," and structural keywords.

The word cloud analysis is qualitative, but it provides a concrete hypothesis for why model optimizations underperform: models are micro-optimizers operating at the implementation level, while experts are architectural optimizers operating at the design level. A model that changes attribute handling or import statements can achieve small constant-factor improvements, but it cannot restructure a data pipeline, introduce caching at the right abstraction boundary, or replace an O(n²) algorithm with O(n log n) — the kinds of changes that produce the 30–80% speedups captured in SWE-Perf's expert patches (recall the maximum performance ratio of 87.8% from Table 1). This hypothesis is consistent with the runtime analysis (Figure 9): micro-optimizations help on short-running functions but cannot address the algorithmic bottlenecks in long-running ones.

Ablation Studies and Robustness Checks

The paper does not include formal ablation studies in the traditional sense (e.g., removing components and measuring performance degradation). This is because SWE-Perf is a benchmark, not a proposed method — the "system" being evaluated is the model, not a novel architecture with ablataable components. However, several implicit ablations and robustness checks emerge from the experimental design:

Oracle vs. Realistic comparison as an ablation of localization: By comparing the same base model (Claude-3.7-sonnet) in the Oracle setting (1.24% Performance) vs. the Realistic setting with OpenHands (2.26% Performance), the paper implicitly ablates the effect of providing perfect localization information. The result — that providing perfect localization reduces performance relative to the agent-based Realistic approach — is a non-obvious finding that challenges the assumption that localization is the primary bottleneck for performance optimization. It suggests that iterative refinement and multi-step reasoning (available in OpenHands but not in single-pass Oracle prompting) provide benefits that outweigh the localization advantage.

Agentless vs. OpenHands as a comparison of architecture: Both systems use the same base model (Claude-3.7-sonnet) and operate in the same Realistic setting. The performance difference — 2.26% vs. 0.41% — is attributable to architectural differences: pipeline-based vs. agent-based, single-pass localization vs. iterative reasoning, and correctness-oriented candidate selection vs. open-ended exploration. This comparison is effectively an architecture ablation, showing that agent-based approaches transfer better to performance optimization than pipeline-based approaches designed for bug fixing.

Correctness-gated Performance (Figures 5, 6) as an ablation of patch quality: By recomputing Performance only on correct instances, the paper ablates the effect of patch-generation failures on aggregate scores. The finding that Performance_pass for top models approaches or exceeds expert levels (OpenAI-o3: 12.2% vs. expert 11.6%; OpenHands: 11.4% vs. expert 11.4%) reveals that the aggregate Performance gap is primarily a correctness problem, not an optimization quality problem — an insight that the aggregate metric obscures.

Model scaling as an implicit capability ablation: The Oracle results (Table 2) evaluate models across a range of scales and architectures — from GPT-4o to Claude-4-opus to OpenAI-o3 to Gemini-2.5-Pro. The compressed Performance range (0.41% to 1.76%) across these very different models suggests that scale alone does not unlock performance optimization capability — moving from smaller/older models to larger/newer ones produces marginal gains, not qualitative improvements. This is consistent with the interpretation that performance optimization requires capabilities (algorithmic reasoning, architectural design sense) not strongly correlated with general LLM capability as measured by standard benchmarks.

Multiple measurement repetitions as a robustness check on runtime data: While not an ablation in the experimental design sense, the paper's measurement methodology — 20 repetitions with IQR outlier filtering and Mann-Whitney U significance testing — serves as an implicit robustness check that the reported Performance values are not artifacts of measurement noise. The δ-computation algorithm systematically weakens observed improvements until the statistical signal disappears, providing a conservative estimate that would be zero if the improvement were purely noise-driven. The fact that the expert achieves 10.85% under this conservative methodology validates that genuine, reproducible improvements exist in the benchmark; the fact that models achieve near-zero on most repositories and runtime bins indicates their improvements are either very small or indistinguishable from noise.

Critical Assessment

The paper's central claim is that SWE-Perf reveals a "substantial capability gap between existing LLMs and expert-level optimization performance" and that this gap points to "critical research opportunities." The experiments genuinely support this claim, but with important qualifications about what "the gap" actually reflects and what conclusions can be drawn.

What the experiments actually demonstrate: The aggregate Performance results (Table 2) show that current LLMs produce statistically significant performance improvements that are, on average, dramatically smaller than those achieved by human experts (2.26% vs. 10.85% for the best system). This is a robust finding that holds across models, architectures, and most repositories. However, the nature of this gap is not uniform — it is driven primarily by two factors that the paper's own analysis reveals:

Correctness failures, not optimization weakness, dominate the aggregate gap. The Performance_pass analysis (Figures 5, 6) shows that on instances where models successfully preserve correctness, their optimization quality can approach or match expert levels. OpenAI-o3 achieves 12.2% Performance_pass vs. expert 11.6%; OpenHands achieves 11.4% vs. expert 11.4%. This means the 8.59-percentage-point gap between OpenHands and Expert in the aggregate (2.26% vs. 10.85%) is not primarily because models produce weak optimizations — it is because models produce patches that fail to apply correctly or break functionality, earning zero Performance on those instances. The paper's strong claim about "optimization capability gap" should therefore be qualified: there is a correctness-preservation gap that manifests as a performance gap in the composite metric, but the underlying optimization reasoning capability appears more developed than the aggregate numbers suggest.
The gap is concentrated in specific difficulty dimensions. Figures 8 and 9 show that models approach expert performance on instances with few target functions (1–2) and short runtimes, but the gap widens dramatically as function count increases or runtime lengthens. This is not merely "models are worse everywhere" — it is "models fail on the instances that require compositional reasoning and algorithmic analysis." The practical implication is that for simple, localized optimizations, models may already be useful; for the complex, high-impact optimizations that matter most in production, they are not.

Genuine weaknesses in the experimental design:

Lack of statistical confidence intervals. The paper reports Performance as point estimates (2.26%, 1.76%, etc.) without confidence intervals or standard errors. Given that Performance is computed from 140 instances with high variance (some repositories contribute near-zero scores, others contribute outlier-high scores), the aggregate numbers may have wide confidence intervals. The claim that OpenHands (2.26%) outperforms Oracle Claude-3.7-sonnet (1.24%) could be within the margin of error given this variance. The paper does not report whether any pairwise differences between methods are statistically significant. Without this, the ranking of methods is suggestive rather than definitive.

The Oracle setting's prompt may systematically disadvantage direct models. The Oracle setting uses a fixed chain-of-thought prompt with a specific output format (SEARCH/REPLACE blocks), while OpenHands can iteratively explore, execute commands, and refine its approach over up to 50 iterations. The comparison is not just "model vs. agent" — it is "single-pass generation with a specific prompt format vs. multi-step autonomous agent." The finding that OpenHands outperforms Oracle does not necessarily mean agent architectures are superior for optimization; it could mean the Oracle prompt format is suboptimal or that multi-step refinement is always beneficial regardless of task. A fairer Oracle baseline would give models multiple attempts, allow iterative refinement, or at minimum ablate different prompt strategies. The paper's implicit claim that the Oracle setting isolates "pure code generation capability" is weakened by the confounding of generation format (single-pass, specific prompt) with generation capability.

No comparison of the same model in an iterative Oracle setting. A natural experiment would be to give the Oracle model multiple refinement passes — either by feeding its own output back as context or by allowing it to revise its patch after seeing test results — and compare to OpenHands. This would distinguish whether the agent advantage comes from iterative refinement (which could be given to any model) or from the specific architecture of OpenHands (code execution, file navigation, etc.). This experiment is not run, making it impossible to determine the source of the Oracle-Realistic performance gap.

The expert patches are the only reference implementation, but they may not be representative of all possible optimizations. The paper acknowledges (Appendix A.1) that expert patches "may not represent the optimal achievable performance." This is important for interpreting model performance: a model that achieves 0% Performance does not necessarily mean it produced no speedup — it could mean its speedup was below the 5% δ threshold, or that its optimization strategy was different from the expert's and did not directly speed up the same tests. The evaluation ties performance measurement to specific tests identified as improvable by the expert patch. A model that dramatically speeds up the codebase but does so through tests not in the expert-identified set would score zero on Performance because Performance is computed only on the pre-identified performance-related tests. This is a reasonable design choice for a benchmark (it provides a consistent evaluation target), but it means Performance is a measure of "did the model improve the specific things the expert improved?" rather than "did the model improve overall codebase performance?" The distinction matters for interpreting near-zero scores: they may partially reflect evaluation scope rather than genuine failure to optimize.

The 140-instance benchmark from 9 repositories is relatively small for drawing general conclusions about LLM optimization capability. With only 54 instances from xarray and 32 from sklearn dominating the benchmark (together 61% of instances), the aggregate Performance scores are heavily influenced by model performance on these two repositories. The finding that OpenHands outperforms the expert on sklearn but dramatically underperforms on xarray suggests that repository-specific characteristics strongly modulate results. Expanding the benchmark to more repositories and a more balanced distribution would be necessary to determine whether current findings generalize or are artifacts of the specific repositories chosen.

The evaluation does not account for the cost of the agent architecture. OpenHands achieves the best Performance (2.26%) but uses up to 50 iterations of reasoning and action per instance — each involving model calls, code execution, and file operations. The Oracle setting uses a single model call. The paper does not compare methods at equal compute budgets, making it impossible to determine whether OpenHands's advantage comes from better architecture or simply from more computation. A FLOPs-matched or dollar-cost-matched comparison would reveal whether the agent advantage is efficiency or simply brute force.

Missing baselines that would strengthen the paper: Several natural baselines are absent:

A retrieval-augmented Oracle baseline: Provide the model with the entire repository but also with retrieval tools (e.g., code search) to see if simple retrieval closes the gap to agent-based methods.
A multi-pass Oracle baseline: Allow the Oracle model to generate a patch, execute tests, see failures, and revise — mimicking the iterative refinement available to OpenHands but without the agent architecture.
A human study baseline: Have human developers (not the original patch authors) attempt to optimize SWE-Perf instances with similar time constraints to model budgets, to establish whether the expert-model gap is specific to LLMs or reflects the general difficulty of optimization for anyone other than the original developer.
A "no optimization" baseline: Measure what happens if the model simply returns the original code unchanged — this would calibrate the δ-computation's behavior under the null hypothesis (no actual change) and verify that the statistical methodology correctly assigns zero gain.

Experiments that would have strengthened the paper:

Ablation of the prompt design. The Oracle prompt (Figure 14) includes specific optimization guidance ("optimization may be achieved either directly through modifications to the objective functions or indirectly by improving computationally intensive subroutines"). Varying this guidance — more specific vs. more open-ended — would test whether the prompt influences optimization strategy and whether models respond differently to different levels of instruction specificity.
Analysis of optimization types attempted. The word cloud analysis (Figures 10–13) is suggestive but coarse. A more systematic analysis — classifying patches into categories like "algorithmic change," "data structure change," "caching addition," "import optimization," "loop restructuring," "constant-factor tweak" — would provide actionable insight into what kinds of optimizations models attempt vs. what experts do. This would directly test the paper's implicit hypothesis that models are micro-optimizers while experts are architectural thinkers.
Sensitivity analysis of the δ threshold. The δ-computation uses α=0.1 and a final threshold of 0.05 for including instances in the benchmark. Varying these parameters and examining how the ranking of methods changes would test whether the reported Performance differences are robust to reasonable alternative choices of statistical parameters.
Temporal analysis of optimization trajectories. For OpenHands, recording what changes the agent makes at each iteration and how performance evolves would reveal whether the agent converges toward genuine optimizations or oscillates between different approaches without making progress. This would distinguish "iterative refinement is helping" from "iterative refinement is just burning compute budget."

Conditional validity of the paper's claims:

The claim that "all models exhibit substantial room for improvement on SWE-Perf" (Section 5.2) is robustly supported by the aggregate Performance numbers — no model approaches the expert's 10.85%, and the 2.26% achieved by the best system is clearly sub-expert. However, the claim that this gap reflects "the complexity of cross-module and repository-scale optimizations" (Section 6) is partially supported: the function-count analysis (Figure 8) shows models degrading on multi-function instances, supporting the cross-module complexity claim, but the Performance_pass analysis (Figures 5, 6) suggests the primary bottleneck is correctness preservation, not cross-module reasoning per se. A model could have excellent cross-module optimization reasoning but fail to implement it correctly, producing the same aggregate results.

The claim that OpenHands "demonstrates superior performance due to its agent-based methodology, providing a flexible and extensible platform for autonomous software development agents" (Section 5.2) is supported directionally but confounded with compute budget: OpenHands does outperform alternatives, but whether this is due to the agent architecture, the higher effective compute budget (50 iterations vs. 1 generation for Oracle), or some interaction cannot be determined from the reported experiments.

The claim that "the model already rivals the performance of the Expert on certain repositories; for instance, on sklearn, OpenHands outperforms the Expert by 0.4%" (Section 5.2) is supported as a point estimate but should be interpreted cautiously: without confidence intervals, a 0.4% difference on 32 instances may not be statistically distinguishable from zero. The paper correctly characterizes this as "an early-stage breakthrough" and "an early sign of potential," which is appropriately hedged.

The claim that models focus on "low-level data structures and basic functionality" while experts emphasize "high-level abstractions and data integrity" (Section 5.3.4) is supported qualitatively by word clouds but is correlational, not causal. The word frequency differences could reflect the repositories or instances where each method succeeded rather than inherent strategy differences — a model that succeeds on xarray (where patches happen to involve attribute handling) would show different word patterns than a model that succeeds on sklearn (where patches may involve type annotations). The paper's word cloud analysis does not control for repository-level confounds.

In summary, the experiments establish that current LLMs underperform human experts on repository-level code performance optimization as measured by statistically rigorous runtime improvement. The nature of this underperformance — whether it is primarily an optimization reasoning problem, a correctness-preservation problem, a retrieval problem, or something else — is partially illuminated by the paper's analyses but not definitively resolved. The benchmark provides a foundation for investigating these questions, and the initial results suggest that the most promising near-term direction is improving correctness preservation for models that already demonstrate latent optimization capability on the instances where they successfully apply patches.

6. Limitations and Trade-offs

Limitation 1: The Difficulty Estimation Pre-Computation Cost Is Unaccounted For, Making the ~2.26% Performance Number an Upper Bound on Deployable Gains

The assumption or constraint. The Realistic setting — and OpenHands as an agent architecture — assumes the system can identify optimization targets autonomously by navigating the repository. But the SWE-Perf evaluation framework provides the system with "target functions" as input even in the Realistic setting: the functions directly invoked by the performance-related unit tests, extracted via yappi dynamic profiling during data collection (Phase 5, Section 3.2). The paper never accounts for the cost of obtaining these target functions at deployment time. Unlike the Oracle setting (where target functions are the ground-truth modified functions), the Realistic setting's targets are "the directly measured functions, not necessarily the ones directly modified; the functions requiring modification might be those it calls" (Section 3.2, Phase 5). This means that even in the "realistic" scenario, the benchmark provides a curated entry point into the optimization problem — the system does not need to discover which functions are performance-relevant, only how to optimize them and their callees.

The consequence. The reported 2.26% Performance for OpenHands (Table 2) is an optimistic upper bound on what the same system would achieve if deployed on a genuinely unseen repository with no pre-profiled targets. In a true zero-shot deployment, the agent would need to: (1) determine which repository operations are performance-critical (which the benchmark pre-computes via profiling), (2) identify which tests measure those operations (which the benchmark pre-identifies via dynamic execution tracing in Phase 3), and (3) establish baseline runtimes to measure improvement (which the benchmark pre-computes via 20-repetition measurement). These steps are computationally expensive — Phase 2 of data collection consumed thousands of CPU-hours running full test suites on tens of thousands of codebases, with single sklearn codebases averaging 83.89 minutes for test execution (Table 4, Appendix B). A deployed system would need to replicate some fraction of this profiling just to establish what "performance" means for a given repository, and the paper provides no guidance on how to do this efficiently or how it would affect the effective budget. The 2.26% headline number therefore reflects performance after someone has already done the expensive profiling work, not the end-to-end cost of autonomous optimization from scratch.

What evidence exists in the paper. Table 4 (Appendix B) documents the extreme cost of Phase 2 test execution: xarray codebases average 58.11 minutes, sklearn averages 83.89 minutes, and sympy averages 24.60 minutes per codebase — and these are only the successful executions out of 19,499 codebases tested. The paper notes that Phase 2 is "the most time-consuming step in the entire data collection pipeline" and that for xarray, "testing a single codebase may take over one hour on a single-core CPU" (Section 3.2, Phase 2, Step 2). The paper also explicitly states that target functions for the Realistic setting are extracted via yappi profiling (Section 3.2, Phase 5) — a step that requires running the performance-related tests with a profiler attached, adding overhead beyond the baseline test execution. No experiment measures how OpenHands or Agentless would perform if asked to identify optimization targets without pre-extracted function lists, nor does any ablation vary the quality or specificity of the target function information provided to the agent.

Mitigation status. The paper does not attempt to address this limitation and does not discuss it in the Limitations section (Appendix A.1). The authors frame the target function provision as a task formulation choice that "restrict[s] the evaluation scope to performance-related tests" (Section 3.1), justified by the prohibitive cost of running full test suites and the challenge of identifying optimization targets in large codebases. They encourage future work to "explore approaches that omit the performance-related unit tests and instead directly optimize the entire codebase" (Section 3.1). This is a reasonable scoping decision for a first benchmark, but it means the benchmark evaluates a scoped version of repository-level optimization — one where the system is told which functions matter — rather than the fully autonomous task the paper's framing sometimes suggests. A practitioner evaluating whether OpenHands could optimize their own repository would need to account for the additional cost of profiling infrastructure that SWE-Perf provides for free.

Limitation 2: Single-Domain Evaluation on 9 Python Repositories With Heavy Concentration in Numerical/Scientific Computing — No Evidence of Cross-Language or Cross-Domain Generalization

The assumption or constraint. SWE-Perf evaluates LLMs exclusively on Python repositories drawn from the scientific computing and systems-tooling ecosystem: xarray (54 instances), scikit-learn (32), sympy (20), astropy (12), sphinx (8), seaborn (6), pylint (3), matplotlib (3), and requests (2). All are Python, all are open-source, and the dominant repositories are numerical/scientific computing libraries where performance optimization typically involves NumPy/SciPy-level array operations, algorithmic complexity in mathematical routines, or data structure choices for large-scale computation. The paper acknowledges this scope limitation explicitly: "the current version of SWE-Perf is constructed from a limited set of open-source repositories, future work could expand the dataset to improve coverage and generalizability" (Appendix A.1).

The consequence. The findings are uninformative about LLM performance optimization capability in other programming languages, other domains, and other optimization patterns. Several specific gaps matter for practitioners:

Compiled languages (C, C++, Rust, Go): Optimization in compiled languages involves different skills than Python optimization — memory layout, cache coherence, SIMD vectorization, and compiler optimization flags — that are largely absent from Python optimization, which focuses on algorithmic complexity, avoiding interpreter overhead, and leveraging C-extensions like NumPy. A model that produces 2.26% gains in Python may perform very differently on C++ codebases where the optimization surface area involves pointer arithmetic, move semantics, or template metaprogramming.
Web/application codebases: The benchmark contains no web frameworks (Django and Flask were in the original 12 repositories but produced zero qualifying instances after Phase 4 filtering, per Table 3 in Appendix B), no front-end code, no database query optimization tasks, and no distributed systems code. Optimization patterns in these domains — connection pooling, query plan optimization, caching strategies, load balancing — are structurally different from the algorithmic and data-structure optimizations that dominate scientific Python libraries.
Performance patterns beyond CPU time: The benchmark evaluates only runtime reduction on CPU-constrained (single-core or 5-core) execution. It does not measure memory optimization, I/O throughput improvement, network latency reduction, GPU utilization, or energy efficiency — all of which are valid and important dimensions of performance optimization in production systems. The paper's focus on runtime is a reasonable scoping choice, but it means "Performance" in this context means specifically "CPU time reduction under the benchmark's measurement protocol," not the broader concept of software performance.
Optimization patterns specific to scientific Python: The dominant repositories (xarray, scikit-learn, sympy) involve domain-specific optimization knowledge — understanding of NumPy broadcasting semantics, familiarity with sparse matrix formats, knowledge of symbolic algebra algorithms — that may not transfer to other domains. The paper's word cloud analysis (Figure 11) shows expert patches emphasizing domain-specific terms like "dtype," "literal," and "sympy," suggesting the optimization strategies are tightly coupled to domain knowledge that models acquire (or fail to acquire) from pretraining data whose coverage varies by domain.

The repository-level performance breakdown (Figure 4) provides direct evidence of this domain-sensitivity: OpenHands achieves 14.5% on sklearn but 0.7% on sympy and 0.2% on matplotlib — a 72× performance ratio between the best and worst repositories. Expert performance also varies (31.8% on xarray vs. 2.6% on seaborn), but the expert-to-model gap is not uniform: it is narrow on sklearn (expert 14.5% vs. OpenHands 14.5%) and catastrophic on xarray (expert 31.8% vs. OpenHands 5.1%). This heterogeneity means the aggregate 2.26% Performance number is meaningless as a general statement about LLM optimization capability — it is a weighted average of near-expert performance on one repository and near-zero performance on others, and the weights are determined by how many PRs happened to survive filtering for each repository rather than by any principled sampling of real-world optimization tasks.

What evidence exists in the paper. Figure 4 provides the per-repository breakdown. Table 1 notes that the 140 instances span only 9 repositories, with 61% concentrated in xarray (54) and sklearn (32). Table 3 (Appendix B) shows that 3 of the original 12 candidate repositories — django, flask, and pytest — produced either zero or near-zero qualifying instances after Phase 4 filtering, meaning the benchmark systematically excludes certain types of repositories even within the Python ecosystem. The paper's Limitations section (Appendix A.1) explicitly acknowledges the repository scope as a primary limitation.

Mitigation status. The paper acknowledges this limitation candidly (Appendix A.1) and frames it as motivation for future dataset expansion. No cross-language, cross-domain, or cross-optimization-pattern experiments are conducted. The mitigation is entirely deferred to future work. For practitioners, this means the paper's findings should be interpreted as "LLM performance optimization capability on scientific Python libraries with pre-identified optimization targets" rather than the broader "repository-level code performance optimization" the title suggests. The strong performance on sklearn (matching expert) is encouraging for the specific domain of machine learning library optimization but cannot be extrapolated to other software engineering contexts without replication.

Limitation 3: The Expert Patches Define the Performance Ceiling, But They Are Not Necessarily Optimal — The Benchmark Cannot Distinguish "Model Is Weak" From "The Optimization Target Is Hard to Discover"

The assumption or constraint. Every SWE-Perf instance is grounded in a real pull request where a human developer achieved a verified, statistically significant performance improvement. The expert patch serves as the gold-standard reference, and model Performance is evaluated by how much runtime reduction the model achieves relative to the original codebase, compared to how much the expert achieved. The paper is transparent that these expert patches are lower bounds on achievable performance, not optimal solutions: "human-written patches... may not represent the optimal achievable performance, potentially underestimating the true upper bound of improvement" (Appendix A.1).

The consequence. This design creates an evaluation asymmetry: models are penalized for producing optimizations that are different from the expert's strategy, even if those optimizations are genuinely performance-improving, because Performance is measured on the specific unit tests that the expert patch accelerated. Consider a scenario where the expert optimized function A (achieving a 40% speedup) by modifying the algorithm in A itself, and a model instead optimizes function B (a callee of A), achieving a 20% speedup on the same test. The model's optimization is genuine and useful, but it would score lower than the expert because the performance test measures the composite runtime of A+B, and the model's 20% improvement on B may translate to only a 5% end-to-end improvement on the test — even if the model's change is arguably more impactful (improving B speeds up all callers, not just A). The benchmark's evaluation structure ties performance measurement to the expert's optimization footprint, implicitly treating the expert's approach as the definition of the optimization task rather than one strategy among many.

A more subtle consequence: the expert patches may reflect optimizations that are relatively discoverable — changes that a developer working on a specific issue or feature happened to notice and implement. These may not be the most impactful optimizations possible, nor representative of the hardest optimization challenges. If expert patches tend to be "low-hanging fruit" (obvious inefficiencies caught during routine development), then the expert baseline may underestimate what skilled, dedicated optimization work could achieve, making the model-expert gap appear smaller than the true capability gap. Conversely, if expert patches reflect deep domain knowledge (restructuring xarray's internal data pipeline, for example), they may represent an unrealistically high bar for general-purpose models — the expert had months or years of context on the codebase, while the model has only what fits in the prompt. The paper provides no characterization of expert patch difficulty or discoverability, making it impossible to calibrate what the Performance numbers mean in terms of optimization challenge.

What evidence exists in the paper. The paper's own data provides suggestive evidence that expert patches are not always optimal: OpenHands outperforms the expert by 0.4% on sklearn (Figure 4), which directly demonstrates that some expert patches left optimization headroom on the table. The Performance_pass analysis (Figures 5, 6) shows that on the subset of instances where models produce correct patches, several models achieve Performance_pass values that match or slightly exceed the expert reference computed on the same subset — OpenAI-o3 achieves 12.2% vs. expert 11.6% (Figure 6), and OpenHands achieves 11.4% vs. expert 11.4% (Figure 5). This reinforces that expert patches are achievable targets, not performance ceilings. However, the paper does not analyze which expert patches models exceed, why those patches had remaining headroom, or whether the model's alternative optimization strategy genuinely outperforms the expert's or simply optimizes a different performance path.

Mitigation status. The paper acknowledges the limitation explicitly (Appendix A.1) but does not attempt to characterize how far expert patches are from optimal or whether the model-expert gap would change if a stronger expert baseline were available. No experiment varies the quality of the reference implementation or tests model performance against multiple alternative optimizations for the same instance. For practitioners, this means SWE-Perf evaluates "can models replicate or approach the specific optimizations that open-source developers happened to contribute?" rather than the more ambitious "can models optimize code to near the performance frontier?" The difference matters: a model that matches the expert on every instance might still leave 2× performance on the table if the expert patches were only modest improvements. The benchmark cannot reveal this because it has no independent measure of the performance frontier — only the expert's achieved improvement, which is a point on the frontier, not the frontier itself.

Limitation 4: The Benchmark Evaluates Only Statistically Significant Runtime Improvement on Pre-Identified Tests — Models That Improve Performance Through Different Mechanisms or on Different Code Paths Score Zero

The assumption or constraint. The Performance metric is computed exclusively on the performance-related unit tests identified during data collection — specifically, tests that satisfied the Phase 3 criteria of exercising patched code segments, not being modified themselves, and showing a statistically significant performance improvement from the expert patch (Phase 4). The evaluation protocol runs only these tests during benchmarking, measuring the statistically significant minimum gain via Algorithm 1. If a model improves performance on code paths not covered by these specific tests — for example, by optimizing a widely-used utility function that speeds up many operations, but the performance-related tests happen not to invoke it heavily — the improvement is invisible to the benchmark. A model could genuinely make the codebase faster in ways that matter for real users but receive a Performance score of zero because the specific tests in the benchmark instance were chosen based on what the expert optimized, not what is performance-critical in general.

The consequence. This evaluation design creates a coverage mismatch between what models might optimize and what the benchmark measures. The paper's task formulation already narrows the evaluation scope (Section 3.1) by providing target functions — the system knows which functions to focus on. But the Performance metric narrows further by measuring improvement only through the lens of tests that the expert happened to accelerate. This double narrowing means that:

A model that optimizes a different bottleneck than the expert — for instance, the expert optimized function A's algorithm while the model optimized function B's data loading, both of which speed up the same end-to-end operation — would be evaluated only on whether B's optimization shows up in tests that were selected because they exercise A's patched code. If the performance-related tests happened to be I/O-bound rather than CPU-bound, B's data loading optimization might not register.
A model that improves overall system performance but not the specific measured code paths — for example, by reducing memory pressure that indirectly speeds up many operations — would score zero on Performance because the effect is diffuse rather than concentrated on the expert-identified tests.
The benchmark cannot detect negative performance side effects — a model could improve the performance-related tests by 20% while inadvertently slowing down 50 other untested code paths by 10%, and the benchmark would report a +20% Performance improvement with no awareness of the regressions. Since only the pre-identified tests are executed during evaluation, the benchmark provides no coverage signal for the rest of the codebase.

The paper's own data collection methodology reveals why this matters: Phase 2 ran all unit tests on all codebases, revealing that a single PR might affect many tests in both positive and negative directions. Phase 3 then filtered for tests that showed substantial, statistically significant improvement and exercised patched code. But during evaluation, only the filtered tests are run. The paper does not measure whether model-generated patches introduce performance regressions on non-targeted tests — a critical concern for any optimization that claims to be "safe" for production use.

What evidence exists in the paper. The paper explicitly describes this evaluation design in Section 4: "we execute all performance-related tests (test_i,j)" on both original and post-patch codebases, where "performance-related tests" are the specific tests identified in Phase 3 and verified in Phase 4. The paper does not report what fraction of total repository tests these performance-related tests represent, nor does it measure performance on non-targeted tests for any model-generated patch. The Correctness metric verifies that performance-related tests still pass, but it does not verify that other tests (not in the performance set) still pass, and it does not measure runtime for any tests outside the performance set. Appendix B, Table 4 shows that full test suite execution is extremely expensive (up to 120 minutes per codebase), which explains why evaluation is scoped to the targeted tests — but the consequence is that the benchmark provides no information about global performance impact.

Mitigation status. The paper does not discuss this limitation. The scoping of evaluation to performance-related tests is presented as a practical necessity driven by the prohibitive cost of running full test suites (Section 3.1, motivation 1). The paper does not acknowledge that this scoping means the Performance metric is a local measure (improvement on targeted code paths) rather than a global measure (improvement on overall codebase performance), nor does it discuss the risk of undetected performance regressions on untested paths. For practitioners, this means SWE-Perf's Performance scores should be interpreted as "improvement on the specific operations the expert optimized" rather than "overall codebase speedup." A model with a 2.26% Performance score may have made the codebase faster, slower, or had no net effect on untested operations — the benchmark provides no signal either way.

The assumption or constraint. The Correctness metric uses a strict logical AND across all performance-related unit tests: an instance counts as correct only if every single performance-related test passes after the patch is applied. This is a binary gate — partial correctness is not rewarded, and the severity of failure is not measured. A model that produces a near-perfect optimization passing 9 of 10 tests receives the same Correctness score (zero for that instance) as a model that breaks all 10 tests. Since Performance is computed with P_i = 0 for any instance that fails Correctness, the Performance metric inherits this all-or-nothing property: a model that achieves expert-level speedup on 90% of instances but breaks one test on each receives an aggregate Performance near zero.

The consequence. The aggregate Performance scores conflate two distinct failure modes that have very different implications for practical utility:

Functional breakage: The model's optimization genuinely changes program behavior in ways that cause test failures. This is a correctness problem — the model produced an incorrect transformation — and improving it requires better verification, more conservative editing, or architectures that explicitly preserve semantics.
Test sensitivity: The model's optimization is functionally correct but changes behavior in ways that the test suite is sensitive to without being genuinely incorrect — for example, changing iteration order in a context where order doesn't matter for correctness but the test expects a specific order, or changing floating-point precision slightly in ways that accumulate to test assertion differences. This is a test fragility problem, and a model that triggers such failures might still be producing useful optimizations for practical purposes.

The Performance_pass analysis (Figures 5, 6, Section 5.3.1) partially addresses this by showing that on the subset of instances where models do achieve correctness, their optimization quality approaches expert levels. But this analysis is purely correlational — it shows that correctness failures are the bottleneck, but it cannot tell us whether the failures are genuine functional breakage or test sensitivity. A model might be producing excellent optimizations that fail tests due to minor behavioral differences that would be acceptable in practice (e.g., slightly different floating-point results within tolerance, different but valid iteration orders), and the benchmark would score these as failures indistinguishable from genuinely broken optimizations.

This binary correctness gate also creates a perverse incentive for conservative optimization: a model that makes only trivial changes (e.g., import caching, minor constant-factor tweaks) is more likely to preserve correctness and achieve a nonzero Performance score, while a model that makes aggressive, high-impact architectural changes is more likely to break something and score zero — even if the architectural changes would, when correct, produce dramatically better performance. The benchmark's design may systematically favor micro-optimizers over architectural optimizers, which is precisely the behavior pattern the word cloud analysis (Section 5.3.4) attributes to current models. The paper cannot determine whether models are micro-optimizers because they cannot think architecturally or because the benchmark penalizes architectural thinking through correctness failures — both explanations are consistent with the data.

What evidence exists in the paper. Table 2 shows substantial Correctness drop-offs from Apply rates: Claude-4-opus drops from 85.71% Apply to 78.57% Correctness, GPT-4o drops from 63.57% to 56.43%, DeepSeek-V3 drops from 47.85% to 42.86%. These drops represent 7–8 percentage points of instances where the model produced an applicable patch but the patch broke at least one test. The paper does not analyze which tests break, how many tests break per failing instance, or whether the breakages follow patterns (e.g., floating-point precision, ordering sensitivity, genuine logic errors). The Performance_pass analysis (Section 5.3.1) demonstrates that the Performance gap is largely attributable to these Correctness failures, but it does not investigate their nature.

Mitigation status. The paper does not address this limitation. The Correctness metric is presented as a straightforward evaluation criterion without discussion of its all-or-nothing property or the conflation of failure modes. The Performance_pass analysis is the closest the paper comes to addressing it — by showing what happens when failures are excluded — but this analysis treats correctness as a filter to be removed rather than a phenomenon to be understood. For practitioners, this means SWE-Perf's scores should be interpreted with awareness that a significant fraction of "failures" may represent optimizations that are functional in practice but trigger test suite sensitivities — and that the benchmark provides no mechanism to distinguish these from genuinely broken optimizations. A model achieving 2.26% Performance might, in a more permissive evaluation (e.g., with human judgment on test failures), achieve substantially higher.

Limitation 6: No Comparison at Equal Compute Budget — OpenHands Uses Up to 50× More Inference Compute Than Oracle Models, Confounding Architecture With Computation

The assumption or constraint. The paper compares methods that use vastly different amounts of inference computation without controlling for compute budget. The Oracle setting uses a single model call per instance — one prompt, one generation. OpenHands uses up to 50 iterations of reasoning and action per instance (Section 5.1), where each iteration involves model calls (potentially multiple), code execution, file operations, and environment interaction. Agentless uses a fixed multi-stage pipeline with hierarchical localization and repair, also involving multiple model calls. The paper reports only the maximum iteration count (50 for OpenHands) and sample number (1 for Agentless) but does not report actual token counts, total model calls, wall-clock time, or dollar cost for any method. There is no experiment or analysis that controls for compute expenditure — no comparison of, say, "OpenHands with 5 iterations vs. Oracle with 5 independent generation-and-select attempts" or "OpenHands with a total token budget matched to Oracle's single generation."

The consequence. The paper's central comparative claim — that OpenHands (2.26%) "demonstrates superior performance due to its agent-based methodology" (Section 5.2) relative to Oracle Claude-3.7-sonnet (1.24%) and Agentless (0.41%) — is uninterpretable as a statement about methodology because compute budget is not controlled. OpenHands could achieve its advantage through any of: (a) the agent architecture enabling better reasoning, (b) simply consuming more computation (more tokens, more model calls), (c) iterative refinement (seeing its own outputs and test results), or (d) some interaction between these. A simple Oracle baseline with 50 independent generation attempts and majority voting on the best-applying patch might match or exceed OpenHands at similar compute cost — but this experiment is not run. Conversely, Oracle models might achieve better results if given the same 50-iteration budget in a simpler iterative-refinement loop without the full agent architecture — but this experiment is also not run.

This compute-confounding problem is particularly acute because the Performance_pass analysis (Figures 5, 6) reveals that on instances where patches succeed, Oracle models can achieve expert-level optimization quality without iterative refinement — OpenAI-o3 achieves 12.2% Performance_pass vs. expert 11.6% (Figure 6). This suggests that the raw optimization capability exists in single-pass generation; what OpenHands's additional compute may be buying is primarily better correctness preservation (through iterative testing and refinement) rather than better optimization ideas. If so, the correct interpretation of the OpenHands advantage is "iterative refinement with test feedback improves correctness preservation" rather than "agent architecture is superior for optimization reasoning" — but without compute-controlled comparisons, these cannot be distinguished.

The practical implication is that a practitioner cannot determine from this paper whether to invest in agent architectures (OpenHands), pipeline architectures (Agentless), or simpler iterative prompting of direct models. The cost-effectiveness ranking — which method delivers the most optimization per dollar of inference compute — is entirely unknown.

What evidence exists in the paper. The paper provides no data on compute consumption: no token counts, no model call counts, no cost estimates, no wall-clock times for evaluation runs. The Oracle setting describes single-pass inference with generation parameters (temperature 0.2, top-p 0.1, max 8192 tokens; Appendix C.1), and OpenHands describes a 50-iteration maximum (Section 5.1), but these numbers are not converted into any common unit. The infrastructure setup is described in Appendix B (two Linux machines with 256 logical CPU cores and 2.0 TiB RAM), but no runtime data is reported for the evaluation phase. The paper's word cloud analysis (Section 5.3.4) and difficulty characterization (Section 5.3) analyze what models do but not how much computation they use to do it.

Mitigation status. The paper does not acknowledge this as a limitation. The absence of compute-controlled comparisons is a methodological gap that could be addressed in future work by: (1) reporting token counts and model call counts for all methods, (2) running methods at multiple budget levels to trace performance-vs-compute curves, and (3) matching total compute across methods for fair comparison. The paper's contribution as a benchmark means these analyses could be performed retrospectively on the same dataset — researchers could evaluate OpenHands with varying iteration limits or Oracle models with varying numbers of independent samples to establish compute-performance tradeoffs. The paper's initial results provide point estimates that motivate such analysis but do not themselves support conclusions about which methodology is "better" independent of compute expenditure.

7. Implications and Future Directions

How This Work Changes the Landscape

SWE-Perf does not introduce a new model, algorithm, or optimization technique. Its contribution is infrastructure and problem definition — a benchmark that creates the possibility of systematic research on a question that was previously unaskable. This is not a paradigm shift in the Kuhnian sense; it is something more foundational: the establishment of a new evaluation regime that makes a previously invisible research problem visible. Before SWE-Perf, the question "can LLMs optimize code performance at repository scale?" had no empirical answer because there was no measurement apparatus capable of producing one. After SWE-Perf, the question has a provisional answer — "not well, but with specific patterns of strength and weakness" — and, more importantly, a framework for tracking progress.

The conceptual reframing is that performance optimization is a distinct reasoning paradigm from correctness-oriented software engineering. The paper is not merely arguing that performance optimization is "harder" than bug fixing in some continuous sense. It is arguing that the two tasks require qualitatively different capabilities — efficiency reasoning, architectural design sense, compositional optimization strategy formation — that are not exercised by existing SWE benchmarks and not naturally developed by models trained primarily on correctness-oriented code data. This reframing matters because it redirects research attention: rather than asking "how can we make models better at SWE-bench?" (which implicitly assumes all SWE tasks share a common capability substrate), the field should ask "what specific capabilities does performance optimization require that correctness-oriented tasks do not, and how can we develop them?" The paper's Oracle/Realistic decomposition, its multi-dimensional difficulty characterization, and its keyword analysis of modification strategies all serve to operationalize this question — they provide tools for diagnosing where the capability gap lies rather than merely measuring its magnitude.

The paper reconciles a latent contradiction in prior work that was never explicitly articulated because the work had not been done. On one hand, code efficiency benchmarks (Mercury, EFFIBENCH, EvalPerf) demonstrated that models can produce efficient implementations of algorithmic problems — suggesting latent performance reasoning capability. On the other hand, the SWE-bench ecosystem demonstrated that repository-level software engineering is challenging even for correctness — suggesting that the complexity of real codebases would overwhelm any optimization capability. SWE-Perf's results synthesize these perspectives: models do possess some optimization reasoning capability (evidenced by Performance_pass scores approaching expert levels on correct patches, Figures 5 and 6, and by OpenHands matching expert performance on sklearn, Figure 4), but this capability is fragile — it degrades rapidly as optimization scope expands (Figure 8), fails to scale to long-running functions (Figure 9), and is frequently derailed by correctness preservation failures (Table 2). The prior perspectives were not contradictory; they were sampling different regions of a difficulty landscape that SWE-Perf maps for the first time.

Which research directions become more attractive, and which become less so, after SWE-Perf:

More attractive: Research on correctness-preserving code transformation — verified refactoring, semantics-preserving optimization, test-driven guardrails for LLM-generated patches. The Performance_pass analysis (Section 5.3.1) reveals that the primary bottleneck for top models is not weak optimization but correctness failures that zero out otherwise-competent optimizations. This suggests that a model equipped with strong verification — static analysis, test generation, or formal semantic preservation checks — could unlock the latent optimization capability that Performance_pass shows exists. Research on combining LLM-based optimization with verification tools becomes a high-priority direction rather than a niche concern.
More attractive: Research on multi-function, compositional optimization reasoning. Figure 8 shows that the model-expert gap widens dramatically as the number of optimization targets increases — models can handle 1–2 function optimizations at near-expert levels but collapse at 16+ functions. This is not a "more data" problem; it is a reasoning architecture problem. Models need to learn how to formulate a coherent optimization strategy that spans multiple functions and modules without losing coherence or introducing inconsistencies. This suggests research on hierarchical planning for code optimization, where a model first identifies a global optimization strategy and then executes localized changes consistent with that strategy — an approach that mirrors how expert developers work but is absent from current single-pass or iterative-refinement agent architectures.
More attractive: Research on difficulty-aware optimization budget allocation. The paper's difficulty characterization (Section 5.3) shows that optimization difficulty varies systematically with function count, runtime magnitude, and repository domain. A system that could estimate optimization difficulty before committing compute — for example, by profiling the codebase and predicting likely gains — could allocate budget adaptively, spending more iterations on high-potential instances and avoiding wasted computation on instances where the model is likely to fail regardless. This connects the SWE-Perf findings to the broader literature on test-time compute scaling and compute-optimal strategy selection, opening a new application domain for those techniques.
Less attractive: Research on more sophisticated search or agent architectures without accompanying verification improvements. The paper shows that even the best agent architecture (OpenHands) achieves only 2.26% aggregate Performance, and that correctness failures — not optimization weakness — dominate the gap. Building more powerful agents that generate more aggressive optimizations without addressing correctness preservation is likely to worsen aggregate Performance by increasing the rate of functional breakage. The bottleneck is not agent capability; it is agent safety. Research that does not address correctness preservation is optimizing a saturated dimension.
Less attractive: Research that relies on function-level code efficiency benchmarks to make claims about "code optimization capability." The paper demonstrates — through the function-count analysis (Figure 8), the runtime analysis (Figure 9), and the repository-heterogeneity findings (Figure 4) — that function-level performance tells you very little about repository-level performance. A model that scores well on Mercury or EFFIBENCH may still fail catastrophically on SWE-Perf because the challenges are structurally different. The field should treat function-level and repository-level optimization as distinct capabilities requiring distinct evaluation. A paper claiming "our model optimizes code" based solely on function-level benchmarks would now face the obvious question: "have you evaluated on SWE-Perf?"
Less attractive: Research that assumes the expert's optimization strategy is the only valid one. The paper's finding that OpenHands outperforms the expert on sklearn (Figure 4) demonstrates that models can discover optimizations human developers missed. Evaluation frameworks that score models by similarity to a reference patch — as is common in some code generation benchmarks — would systematically penalize valid alternative optimization strategies. SWE-Perf's measurement-based evaluation (runtime, not patch similarity) provides a template for how to evaluate open-ended optimization tasks without constraining the solution space to the expert's approach. This should encourage a shift from "did the model reproduce the reference solution?" to "did the model achieve a comparable or better performance outcome?"

The magnitude of the shift is bounded by the benchmark's scope — Python only, scientific computing dominant, 140 instances — but the direction is clear. SWE-Perf does for code performance optimization what SWE-Bench did for repository-level bug fixing: it creates a shared evaluation target that enables the field to measure progress, compare methods, and diagnose failure modes. The initial results are sobering — models are far from expert-level — but the benchmark's value is not in the results themselves but in the research program they enable. The paper transforms "can LLMs optimize code performance?" from a qualitative debate with anecdotal evidence into a quantitative research question with a reproducible measurement methodology, and that transformation is the landscape change, regardless of what the answer turns out to be as models improve.

Follow-Up Research This Work Enables

1. Training a lightweight difficulty predictor to enable adaptive optimization budget allocation. The most immediate bottleneck for deploying anything like the compute-optimal strategies the paper's difficulty analysis implies is the cost of estimating optimization difficulty. The paper shows that difficulty varies systematically — models succeed on 1–2 function, short-runtime instances and fail on multi-function, long-runtime ones — but extracting these difficulty features currently requires expensive profiling (running performance tests, measuring runtimes) that undermines the efficiency gains of adaptive allocation. A natural follow-up would train a classifier or regression model to predict optimization difficulty directly from static code features — function count from the call graph, cyclomatic complexity, code size, repository domain, dependency structure — without executing a single test. Training data exists implicitly in SWE-Perf: each instance has a known expert-achievable improvement (the performance ratio from Table 1), and models achieve known Performance scores on each instance. A difficulty predictor trained on these features could estimate, for a new unseen instance, the likely Performance a given model would achieve, enabling the system to decide whether to invest computation in optimization or flag the instance for human review. The key experiment: compare an adaptive system that uses predicted difficulty to allocate a fixed total compute budget across instances against a uniform-allocation baseline, measuring aggregate Performance per FLOP or dollar across the full benchmark. The paper's finding that models plateau on instances with long runtimes (Figure 9) suggests that avoiding wasted computation on these instances could substantially improve cost-efficiency even without improving per-instance optimization quality.

2. Combining LLM-based optimization with verified refactoring or symbolic execution to address the correctness-preservation bottleneck. The Performance_pass analysis (Figures 5, 6) reveals that top models achieve expert-level optimization quality on instances where they preserve correctness — OpenAI-o3 achieves 12.2% vs. expert 11.6%, OpenHands achieves 11.4% vs. expert 11.4% — but their aggregate Performance is dragged down by correctness failures on roughly 15–25% of instances (Table 2 Correctness rates range from 56.43% for GPT-4o to 83.57% for Gemini-2.5-Pro). This suggests a natural architectural combination: let the LLM propose optimizations freely, then use a verified refactoring tool (e.g., a semantics-preserving code transformation engine, or a symbolic execution framework like crosshair or deal-solver) to check whether the proposed transformation is behaviorally equivalent to the original on all inputs, not just the test suite. Transformations that pass the verification are applied; those that fail are rejected or sent back for revision. This decouples optimization creativity (where LLMs show latent strength) from correctness verification (where formal methods are reliable), addressing the primary failure mode without constraining the model's optimization strategy. The concrete experiment: evaluate OpenHands augmented with a verification step — after each proposed edit, run the edit through a symbolic execution checker that attempts to find counterexamples where the optimized function produces different output than the original. Measure whether this filtering improves aggregate Performance by rescuing optimizations that would otherwise be rejected for Correctness failures, and whether it degrades optimization quality (false rejections of valid optimizations). The paper's finding that OpenHands already achieves 77.86% Correctness (Table 2) provides a baseline against which to measure the tradeoff between verification strictness and optimization throughput.

3. Developing and evaluating models fine-tuned specifically on performance-improving code transformations, using SWE-Perf's expert patches as training data. The paper's word cloud analysis (Section 5.3.4) reveals that models and experts make qualitatively different kinds of changes — models focus on "low-level data structures and basic functionality," while experts emphasize "high-level abstractions and data integrity." This suggests a training data mismatch: models have seen far more code that tweaks implementation details than code that restructures architecture for performance. SWE-Perf provides 140 expert-authored patches that represent genuine, verified performance optimizations across diverse repositories, each with before-and-after code contexts. These could serve as supervised fine-tuning data to teach models what expert-level optimizations look like — not just the syntactic patterns (which functions to modify, what changes to make) but the semantic reasoning (why this algorithmic change, why this caching strategy, why this data structure substitution). The specific experiment: fine-tune a base model (e.g., DeepSeek-V3 or Qwen3-235B, both evaluated in the paper) on the 140 SWE-Perf expert patches with appropriate train/validation splits (the paper's 140 instances would need to be split, or additional instances collected using the same pipeline from other repositories), then evaluate the fine-tuned model against the original on SWE-Perf's held-out instances using the full three-tier metric hierarchy. Measure whether fine-tuning shifts the model's optimization strategy — via word cloud analysis of generated patches — toward the expert's high-level abstraction patterns, and whether this shift correlates with improved Performance. A negative result (fine-tuning changes patch vocabulary but not aggregate Performance) would suggest the gap is architectural rather than data-driven; a positive result would suggest that expert demonstrations can partially close the gap.

4. Extending the SWE-Perf collection pipeline to compiled languages and performance patterns beyond CPU time, creating a multi-dimensional optimization benchmark. The paper's primary limitation — acknowledged explicitly in Appendix A.1 — is the narrow domain coverage: 9 Python repositories dominated by scientific computing, measuring only CPU runtime under single/few-core constraints. A natural follow-up would replicate the data collection pipeline for repositories in compiled languages (C, C++, Rust, Go) and for additional performance dimensions: memory allocation (peak and average), I/O throughput (disk and network), GPU utilization, and energy consumption (using hardware performance counters or platform-specific tools like perf or nvprof). The pipeline methodology — mine PRs with performance-related keywords, measure before/after in Docker containers with resource constraints, apply statistical filters, extract optimization targets — is language-agnostic and could be adapted with language-specific tooling. The concrete experiment: apply the SWE-Perf collection methodology to the top 10–20 C++ repositories on GitHub (or the equivalent high-star C++ projects) and measure whether the performance ratios, expert-model gaps, and difficulty characteristics observed in Python replicate in compiled-language contexts. The paper's finding that models excel at micro-optimizations but fail at architectural changes (Figures 9–11) would predict even larger gaps in C++, where micro-optimizations (loop unrolling, inlining) are often handled by compilers and human experts focus on memory layout, cache coherence, and parallelization — areas where LLMs likely have even less training signal. A negative result (models performing better on C++ than Python) would suggest the performance gap is domain-specific rather than fundamental, with important implications for which software ecosystems benefit most from LLM optimization assistance.

5. A systematic study of optimization strategy classification to test the "micro-optimizer vs. architectural thinker" hypothesis. The paper's word cloud analysis (Section 5.3.4) provides suggestive evidence that models and experts operate at different levels of abstraction — models tweak attributes and imports while experts restructure data pipelines and algorithms — but the analysis is qualitative and correlational. A rigorous follow-up would develop a taxonomy of optimization types (e.g., algorithmic change (O(n²) → O(n log n)), data structure substitution, caching addition, loop restructuring, import optimization, constant-factor tweak, parallelization, lazy evaluation, memory pre-allocation, API call reduction) and manually classify a sample of expert patches and model-generated patches into these categories. This classification would test specific hypotheses: (1) Do expert patches contain a higher proportion of algorithmic changes and data structure substitutions than model patches? (2) Do model patches achieving high Performance_pass contain different optimization-type distributions than low-Performance_pass patches? (3) Does the optimization-type distribution shift as model scale increases (e.g., from GPT-4o to Claude-4-opus to OpenAI-o3)? The key experiment: if models are genuinely incapable of algorithmic reasoning (as opposed to merely disincentivized by the benchmark's correctness penalties), then no amount of scaling or prompting should increase the proportion of algorithmic changes in model-generated patches, even on instances where the expert patch is demonstrably algorithmic. If scaling does increase algorithmic changes, the capability is latent and the bottleneck is elsewhere (prompt design, evaluation incentives, training data). The paper's existing data — 140 expert patches and model-generated patches from 13 method/model combinations — provides sufficient material for an initial classification study without additional data collection.

6. A negative-result study examining whether the ReST-style self-improvement loop that failed for revision models in other domains (referenced in the paper's Related Work discussion of SWE-Smith and SWE-Gym) also fails for performance optimization, or whether performance optimization's different structure enables successful bootstrapping. The paper discusses prior work on training software engineering agents with self-generated data (SWE-Gym, SWE-Smith) but does not itself attempt to use model-generated optimizations to improve the model. A critical follow-up — and a stress test of the paper's core claim that performance optimization is qualitatively different from correctness-oriented SWE — would be: can a model improve its SWE-Perf Performance by training on its own successful optimizations? The experiment: take a base model (e.g., Claude-3.7-sonnet), run it on SWE-Perf instances, identify instances where it achieves nonzero Performance (i.e., patches that apply, preserve correctness, and achieve statistically significant speedup), use those patches as additional training data (perhaps alongside the expert patches), fine-tune, and re-evaluate. If self-training improves Performance, it suggests that optimization capability can be bootstrapped from weaker models — an important positive result for scalable deployment. If self-training degrades or fails to improve Performance — analogous to the negative revision-model results the paper notes in other domains — it suggests that model-generated optimizations contain spurious patterns or shallow heuristics that, when amplified through training, reduce rather than enhance genuine optimization reasoning. A negative result would be equally informative, establishing a boundary on self-improvement approaches and motivating research on quality filters or expert-in-the-loop training for performance optimization.

Practical Applications and Downstream Use Cases

1. Prioritizing human optimization effort through model-based triage in large-scale codebase maintenance. Organizations maintaining large codebases with thousands of potential optimization targets face a resource allocation problem: which functions should an expensive human expert spend time optimizing? The paper's difficulty characterization (Section 5.3) provides a direct operational template. For a given codebase, an organization could: (a) profile the codebase to identify functions with high cumulative runtime (using the Phase 5 yappi-based methodology from the paper), (b) run a model like OpenHands on each candidate, measuring the statistically significant minimum gain via Algorithm 1, and (c) route instances where the model achieves at least 5% Performance (the paper's inclusion threshold) to automated optimization, while flagging instances where the model achieves near-zero gain — particularly multi-function, long-runtime instances that Figure 8 and Figure 9 show are beyond current model capability — for expert attention. This triage system leverages the model as a "quick optimizer" for the low-hanging fruit (1–2 function, short-runtime optimizations where Figure 8 shows OpenHands approaches expert performance) while reserving scarce human expertise for the high-impact, structurally complex optimizations that models cannot yet handle. The paper's repository-heterogeneity findings (Figure 4) suggest this triage would need to be calibrated per-repository — models perform well on sklearn but poorly on sympy — but the measurement methodology for determining per-instance model capability is exactly what SWE-Perf provides. The key numbers: OpenHands achieves 14.5% on sklearn (Figure 4), matching expert, meaning sklearn optimization instances could be fully automated with current technology; on xarray, the gap is 5.1% vs. 31.8%, meaning human expertise remains essential for the highest-impact optimizations in that repository.

2. Automated performance regression detection in CI/CD pipelines with model-based optimization proposals. Continuous integration pipelines typically run test suites to detect functional regressions, but performance regressions — patches that slow down the codebase without breaking tests — are harder to detect and usually require dedicated benchmarking infrastructure that most projects lack. The SWE-Perf measurement methodology — warm-up runs, 20-repetition measurement, IQR outlier filtering, and δ-computation (Algorithm 1) — provides a turnkey protocol for adding statistically rigorous performance regression detection to any CI pipeline that already runs tests in Docker containers. The extension to optimization: when a performance regression is detected (a PR causes a statistically significant slowdown on a performance-sensitive test), the system could automatically invoke a model like OpenHands with the regression-inducing PR as context, asking it to propose an optimization that recovers or exceeds the original performance. This creates a closed loop: detect slowdown → propose fix → verify improvement → merge or flag. The paper's Correctness metric (Table 2) shows that even the best model fails to preserve correctness on 22.14% of instances (OpenHands Correctness 77.86%), which is too high for fully automated merging. But as a "propose, don't apply" assistant — generating candidate optimizations that a human developer reviews and tests — the system could substantially reduce the time from performance regression detection to resolution. The key numbers: the paper's Phase 4 methodology can detect performance improvements as small as 5% with statistical confidence (the δ inclusion threshold), and the expert patches achieve average gains of 10.85% (Table 2). A CI system using this methodology could detect and propose fixes for regressions of similar magnitude within a single development cycle.

3. Curriculum design for software engineering education and interview assessment using real-world optimization tasks. The SWE-Perf instances provide 140 concrete, verified, real-world code performance optimization exercises — each with a known-before optimization state, a known-after expert-achieved state, and a set of performance tests that quantify the improvement. This is directly usable as educational material: students can be given the original codebase and target functions (the Oracle setting), asked to produce patches that improve performance while preserving correctness, and evaluated against both the expert's achieved improvement and the model baselines reported in the paper. The multi-dimensional difficulty characterization (Section 5.3) enables progressive curriculum design — start with single-function, short-runtime instances where students can achieve meaningful gains with localized changes, then progress to multi-function, long-runtime instances that require architectural thinking. The paper's word cloud analysis (Figures 10–11) and the implicit optimization taxonomy it suggests (micro-optimizations vs. architectural changes) could be formalized into explicit learning objectives: "identify when a performance bottleneck requires an algorithmic change vs. a constant-factor tweak," "trace performance impact across function call boundaries," "verify correctness after cross-module refactoring." For technical interviews, SWE-Perf instances provide standardized, real-world optimization tasks with quantitative scoring rubrics, avoiding the artificiality of typical coding interview problems while providing objective performance criteria. The key numbers: instances span from ~0.01-second runtimes to 25.2-second runtimes (Table 1), with optimization ratios from 10.9% to 87.8%, providing a wide range of difficulty levels for progressive skill development. The 140 instances across 9 repositories provide sufficient diversity for a semester-long course or an interview question bank.

4. Benchmarking and selection of LLMs for internal code optimization tools in large software organizations. Organizations developing internal LLM-based coding assistants face a model selection problem: which of the dozen-plus frontier models (or fine-tuned variants) is best suited for suggesting performance improvements during code review? The paper's evaluation of 10 models under the Oracle setting and 2 systems under the Realistic setting (Table 2) provides a starting point, but organizations could replicate the evaluation protocol on their own repositories using the SWE-Perf data collection pipeline (Phases 1–5, Section 3.2) to create a custom benchmark reflecting their specific codebase characteristics, performance patterns, and optimization priorities. For example, an organization primarily working with numerical Python code might weight sklearn and xarray instances heavily; a web services company might extend the pipeline to their own repositories (Flask, Django, FastAPI codebases) and measure API endpoint latency rather than unit test runtime. The statistically rigorous measurement methodology — particularly Algorithm 1's pessimistic δ-computation — ensures that model comparisons are based on genuine, reproducible optimization capability rather than measurement noise, which is essential for procurement decisions involving significant inference compute expenditure. The paper's finding that model ranking is not uniform across repositories (Figure 4: OpenHands dominates on sklearn but is near-zero on sympy) underscores the importance of repository-specific evaluation rather than relying on aggregate benchmark scores. The key numbers: the paper's collection pipeline filtered 102,241 PRs to 140 instances (0.14% yield), and Phase 2 test execution cost thousands of CPU-hours (Table 4). An organization replicating this for a single repository would face proportionally lower costs, making custom benchmark creation feasible for well-resourced engineering teams. The resulting custom benchmark provides a decision tool: given the organization's actual codebase and performance patterns, which model (and which architecture — direct prompting, pipeline, or agent) delivers the best optimization-per-dollar for their specific context?

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving2025· evaluation← LLM-judged similar
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?2025· evaluation← in-domain predecessor
The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements2025· code← in-domain predecessor
- CASS: Nvidia to AMD Transpilation with Data, Models, and Benchmark2025· code← in-domain predecessor
Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases2025· code← similar topic
- UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2025· agents← LLM-judged similar
- AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents2024· agents← LLM-judged similar
- AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimization2025· code← in-domain predecessor
Large Language Models for Compiler Optimization2023· code← similar topic
- StarCoder: may the source be with you!2023· code← in-domain predecessor
- View publication2024· code← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

SWE-Universe: Scale Real-World Verifiable Environments to Millions2026· agents→ LLM-judged similar
- SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue2026· agents→ in-domain predecessor
- CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents2026· agents→ similar topic
MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios2026· agents→ LLM-judged similar
- Proact-VL: A Proactive VideoLLM for Real-Time AI Companions2026· agents→ in-domain predecessor
- MAI-UI Technical Report: Real-World Centric Foundation GUI Agents2025· agents→ similar topic
- TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios2026· agents→ similar topic
- WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation2026· agents→ similar topic
"TODO: Fix the Mess Gemini Created": Towards Understanding GenAI-Induced Self-Admitted Technical Debt2026· code→ LLM-judged similar
- Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model2026· code→ in-domain predecessor
- A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.52026· safety→ topic thread
Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference2025· code→ in-domain predecessor
- RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation2025· code→ in-domain predecessor
Large Language Models for Compiler Optimization2023· code→ similar topic
- Magicoder: Source Code Is All You Need2023· code→ in-domain predecessor
Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases2025· code→ similar topic
- X-Coder: Advancing Competitive Programming with Fully Synthetic Tasks, Solutions, and Tests2026· code→ in-domain predecessor