Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving

🎯 Pitch

Current LLMs can resolve over 65% of Python issues on SWE-bench Verified, but their performance collapses to below 23% for Java and struggles even more with languages like C and Go, revealing a stark overfitting to Python ecosystems. The study further shows that models cannot handle any issue requiring more than 15 minutes of human effort, with resolution rates dropping to near-zero for complex fixes spanning multiple files or large patch edits.

1. Executive Summary

This paper introduces Multi-SWE-bench, a multilingual benchmark for issue resolving that extends the SWE-bench paradigm across seven non-Python programming languages—Java, TypeScript, JavaScript, Go, Rust, C, and C++—comprising 1,632 human-verified instances curated through a five-phase pipeline involving 68 annotators. The authors evaluate nine frontier LLMs (including GPT-4o, Claude-3.7-Sonnet, DeepSeek-V3, and Qwen2.5-72B-Instruct) using three representative methods—MagentLess (a fixed-workflow approach adapted from Agentless), MSWE-agent (an agent-based approach adapted from SWE-agent), and MopenHands (an interactive agent adapted from OpenHands)—and find that all methods exhibit a substantial performance drop from Python to other languages, with resolved rates on Java reaching at most ~23% (Claude-3.7-Sonnet + MSWE-agent) despite Python rates exceeding 50%. Performance is sharply modulated by human-labeled difficulty, with hard issues (>1 hour of estimated human effort) showing near-zero resolved rates across all models and languages, establishing that current LLM-based agents can only handle tasks solvable by humans in under 15 minutes and remain insensitive to complex, multi-file reasoning demands that require over 600 tokens of patch edits or span multiple files.

2. Context and Motivation

The Core Gap: We Only Know How LLMs Resolve Python Issues

The fundamental problem this paper addresses is deceptively simple: does the impressive progress of LLMs on automated software engineering tasks in Python translate to other widely used programming languages? Since the release of SWE-bench (Jimenez et al., 2023), the field has witnessed a dramatic acceleration in LLM-based issue resolving—the task of taking a GitHub issue description and the corresponding buggy codebase, then producing a patch that fixes the issue. Resolved rates on SWE-bench Verified, a curated subset of 500 Python issues, climbed from essentially zero (0.40% for RAG + GPT-3.5) to 65.40% (Augment Agent v0) in under a year. This trajectory has generated substantial excitement about fully autonomous AI programmers.

However, a critical assumption underlies this entire line of work: that Python is a sufficient proxy for evaluating software engineering capability. Real-world software development spans a vast ecosystem of languages—Java powers enterprise backends, TypeScript and JavaScript dominate web development, Go and Rust underpin cloud infrastructure and systems programming, and C and C++ remain the backbone of performance-critical and embedded systems. Each of these languages imposes distinct demands on an automated agent:

Programming paradigms: Python is predominantly imperative with optional object-oriented features. Java enforces strict class-based OOP. Rust introduces ownership semantics and borrow checking. C and C++ require manual memory management. These aren't syntactic differences—they represent fundamentally different models of computation that a repair agent must understand.
Idiomatic patterns: The "right" way to structure code varies enormously between languages. An agent that generates Pythonic fixes may produce unidiomatic, rejected patches in Go or Rust, where community conventions around error handling, concurrency, and code organization are rigidly enforced.
Runtime behaviors: Python's interpreted execution with dynamic typing makes certain classes of bugs immediate and visible. In contrast, memory safety violations in C/C++ may manifest nondeterministically, type errors in Java may be caught at compile time rather than runtime, and asynchronous execution in JavaScript/TypeScript introduces timing-dependent failures invisible in stack traces. Fixing these bugs requires reasoning about execution models that share little with Python's.
Build systems and tooling: Python projects often use simple setup.py or pyproject.toml configurations. Java projects involve Maven or Gradle with complex dependency resolution. C/C++ projects require Make, CMake, or custom build scripts with platform-specific compilation flags. An agent that navigates Python tooling effortlessly may be completely unable to build, test, or patch a Go or C++ repository—not because it can't understand the bug, but because it can't interact with the environment.

The absence of multilingual benchmarks means the field has been evaluating LLMs on the easiest subset of the real-world software engineering problem. This is not merely an academic gap—it has direct practical consequences for anyone hoping to deploy LLM-based coding agents in production environments.

Why This Gap Matters Now

The timing of this work is not coincidental. Several converging trends make the Python-only evaluation paradigm increasingly untenable:

1. The deployment pipeline is maturing. As LLM-based coding agents move from research demonstrations toward production deployment (e.g., GitHub Copilot's agent mode, Cursor's autonomous features, SWE-agent deployments in CI/CD), the assumption that Python performance generalizes has direct financial and reliability implications. An enterprise considering automated issue resolution for their Java or TypeScript codebase needs evidence, not extrapolation.

2. The RL scaling narrative demands realistic environments. Recent breakthroughs in reasoning models—DeepSeek-R1 (Guo et al., 2025), OpenAI-o1 (Jaech et al., 2024), and o3 (OpenAI, 2025)—have demonstrated that reinforcement learning with verifiable reward signals can dramatically improve LLM performance on complex tasks. The natural next step is applying these techniques to software engineering, where correctness can be verified through test execution. However, RL requires environments: containerized codebases with reproducible execution, test suites, and ground-truth fixes. The Python ecosystem has these (via SWE-bench). The rest of the software world does not—or did not, prior to this work. The Multi-SWE-RL community launched alongside this paper is a direct response to this bottleneck: without multilingual RL training data, the scaling narrative remains Python-centric by necessity.

3. The SWE-bench ceiling is approaching. With resolved rates on SWE-bench Verified reaching 65.40%, the benchmark's discriminatory power may be diminishing. As models saturate Python issue resolving, the field needs harder evaluation surfaces that expose genuine capability limitations rather than Python-specific optimization. Multi-SWE-bench's substantially lower resolved rates—even the best model (Claude-3.7-Sonnet) achieves only 23.44% on Java versus 45.80% on Python—demonstrate that the benchmark provides substantial headroom for measuring progress.

4. Language diversity reflects real-world software engineering. The TIOBE Index, Stack Overflow Developer Survey, and GitHub Octoverse consistently show that Python represents a fraction of global software development. Java, JavaScript, TypeScript, C++, and C# collectively dominate production codebases. A benchmark that excludes these languages evaluates LLMs on an unrepresentative sample of the tasks they would face in deployment.

Where Existing Benchmarks Fall Short

The paper situates itself relative to a progression of code-related benchmarks, each addressing some dimension of software engineering but collectively leaving the multilingual issue-resolving gap unfilled:

Monolingual program-level benchmarks (HumanEval, MBPP, APPS) evaluate whether a model can write a single function given a natural language specification and a handful of test cases. These benchmarks were transformative when introduced—they provided the first standardized evaluation of code generation—but they capture only a narrow slice of software engineering. An issue resolver must understand an entire repository's structure, identify which of thousands of files contains the relevant code, produce a patch consistent with existing codebase conventions, and verify that the fix doesn't break other functionality. Function-level benchmarks test writing; issue-resolving benchmarks test engineering.

Multilingual program-level benchmarks (Multilingual-HumanEval, HumanEval-X, MBXP) addressed the language diversity problem for function generation by translating existing Python benchmarks into multiple languages. This was a step forward, but it inherited the same fundamental limitation: generating a single function in isolation bears little resemblance to navigating a 700,000-line codebase to fix a bug introduced across five files and three abstraction layers.

Repository-level benchmarks expanded the scope beyond individual functions but remained largely Python-focused. RepoBench and CrossCodeEval address cross-file code completion. RepoFixEval and GitBug-Actions target repository-level program repair. SWT-bench evaluates real-world bug fixes. These works progressively increased realism but, with the notable exception of SWE-Lancer (discussed below), concentrated on Python.

SWE-bench (Jimenez et al., 2023) was the breakthrough that defined the issue-resolving task as the field now understands it: given a GitHub issue and a snapshot of the repository at the commit before the fix, generate a patch that causes the associated tests to pass. SWE-bench's 2,294 instances from 12 Python repositories provided the first large-scale evaluation surface for autonomous software engineering. SWE-bench Verified, a 500-instance subset with human-validated test coverage and issue descriptions, became the de facto standard metric. The rapid progress on this benchmark—from near-zero to 65.40% in under a year—both demonstrated the power of LLM-based agents and raised the question of whether the benchmark was capturing generalizable software engineering capability or Python-specific optimization.

SWE-bench Multimodal and Visual SWE-bench extended the evaluation to systems requiring visual reasoning (e.g., frontend bugs manifesting as incorrect rendering), which broadened the task scope but remained within the Python-centric ecosystem.

SWE-Lancer (Miserendino et al., 2025) comes closest to filling the gap this paper addresses. It features over 1,400 freelance software engineering tasks from Upwork, spanning JavaScript and TypeScript. This is the only prior work to systematically evaluate issue resolving in non-Python languages, and it demonstrated that LLMs struggle substantially more on these tasks than on Python equivalents. However, SWE-Lancer focuses specifically on web development languages and the freelance task ecosystem, leaving Java, Go, Rust, C, and C++ entirely unexplored. Its tasks are also sourced from Upwork rather than open-source repositories, which introduces different characteristics: freelance tasks may be smaller in scope, more self-contained, and less representative of the complex, multi-file refactoring required in large production codebases.

How This Paper Positions Itself

Multi-SWE-bench positions itself as a complementary extension of the SWE-bench paradigm rather than a replacement. The paper explicitly adopts SWE-bench's task formulation, evaluation methodology, and annotation standards (aligning with SWE-bench Verified's guidelines). The contribution is not a new task definition but rather demonstrating that the existing task definition generalizes poorly across languages and providing the infrastructure to measure that generalization gap.

The positioning has several key aspects:

1. From Python-centric to truly multilingual. The jump from SWE-bench's 12 Python repositories to Multi-SWE-bench's 39 repositories spanning 7 languages is not merely additive—it changes the nature of the evaluation. A model that performs well on Python can no longer be described as "good at issue resolving"; it must be described as "good at Python issue resolving," with Multi-SWE-bench providing evidence for or against broader competence.

2. Maintaining SWE-bench's rigor at scale. The five-phase construction pipeline (repository selection → PR crawling → environment determination → PR filtering → manual verification) is intentionally aligned with SWE-bench's methodology but adapted for multilingual challenges. The 68 annotators, dual annotation with cross-review, and 80% accuracy threshold for outsourced annotations represent a substantial investment in benchmark quality. The paper emphasizes this because multilingual benchmarks risk lower quality—each additional language introduces new tooling, testing frameworks, and failure modes that are hard to standardize.

3. Bridging evaluation and RL training. The simultaneous release of Multi-SWE-bench (for evaluation) and Multi-SWE-RL (for training) reflects a strategic positioning: the paper argues that both components are necessary for progress. Multi-SWE-bench without training data would identify a gap without providing the means to close it. Multi-SWE-RL without a rigorous evaluation benchmark would risk reward hacking. Together, they form an infrastructure that the authors explicitly frame as a foundation for "scaling RL in real-world software environments" toward AGI—a notably ambitious framing that positions issue-resolving as a path to general intelligence rather than just an engineering automation problem.

4. Emphasizing open-source community contribution. Unlike most benchmark papers that release static datasets, Multi-SWE-bench is designed as a living resource with rolling quarterly updates, contribution guidelines, and incentive plans. This reflects a practical recognition: building comprehensive multilingual benchmarks is too large a task for any single research group. The 4,723 instances in Multi-SWE-RL (without manual verification) represent the output of an automated pipeline that the community can extend to any GitHub repository. The 1,632 manually verified instances in Multi-SWE-bench represent the quality-controlled evaluation subset. This two-tier architecture—automated breadth for training, manual verification for evaluation—is a pragmatic response to the scaling challenges of multilingual benchmark construction.

The Unspoken Tension: Are We Evaluating the Models or the Methods?

A subtle but important tension runs through the paper's positioning. The evaluative framework tests three methods (MagentLess, MSWE-agent, MopenHands) that were originally designed for Python and then adapted for multilingual use through "key modifications" described in Section 5.1. These modifications—revised prompts, truncated observations, .gitignore filtering, bug fixes for tab character rendering—are practical engineering accommodations, but they raise a question the paper acknowledges without fully resolving: are the observed cross-language performance drops due to fundamental LLM limitations, or due to suboptimal adaptation of Python-first methods?

This ambiguity is not a flaw in the paper's design—it reflects the genuine state of the field. No agent designed from scratch for multilingual issue resolving exists, because no multilingual benchmark existed to evaluate one. Multi-SWE-bench enables the development of such agents by providing the evaluation surface. The current results should be understood as establishing a lower bound on multilingual capability: even with reasonable adaptations, Python-optimized agents struggle severely outside Python. Whether language-native agent designs could close that gap is an empirical question that Multi-SWE-bench now enables the community to answer.

Summary of the Motivation

The paper's motivation can be compressed to a single question: Is the progress on SWE-bench evidence of general software engineering capability, or evidence of Python-specific optimization? Multi-SWE-bench provides the measurement instrument to answer that question, and the answer—based on the evaluation of nine frontier models across three methods—is that current LLM-based agents are substantially Python-specialized, with resolved rates on other languages falling by factors of 2× to 20× depending on the language and difficulty level. This finding has immediate implications for deployment decisions, research prioritization, and the RL training data bottleneck that the Multi-SWE-RL community is designed to address.

3. Technical Approach

3.1 Reader Orientation

Multi-SWE-bench is not a single system but a benchmark infrastructure—a dataset of 1,632 real-world GitHub issue-resolving tasks spanning seven programming languages, each packaged with a reproducible Docker environment, executable test suite, and ground-truth fix patch. The core problem it addresses is that the field of LLM-based software engineering has no way to measure whether Python issue-resolving performance generalizes to other languages; the solution takes the form of a five-phase construction pipeline that transforms raw GitHub pull requests into rigorously verified evaluation instances, coupled with adapted versions of three existing agent methods to establish baseline measurements.

3.2 Big-Picture Architecture (Diagram in Words)

The infrastructure comprises five major components:

Repository Selection Pipeline (Phase 1): Filters GitHub repositories by popularity (>500 stars), CI/CD support, and build viability to produce a candidate pool of 39 production-quality codebases across seven languages.
Pull Request Crawling and Filtering Pipeline (Phases 2–4): Crawls all PRs from selected repositories, filters for issue-linked PRs that modify test files, builds Docker containers per-PR, and performs automated semantic validation by running test suites under three patch configurations to identify PRs with clear bug-fixing effects and no regressions. Outputs 2,456 candidate instances.
Manual Verification Pipeline (Phase 5): 68 annotators independently label each candidate against the SWE-bench Verified rubric; dual annotation with cross-review and quality assessment (≥80% accuracy threshold by 14 internal engineers) filters candidates to 1,632 high-quality instances.
Multi-SWE-RL Dataset (Community Infrastructure): An automated pipeline (Phases 1–4 without manual verification) produces 4,723 containerized instances across 76 repositories for RL training, released alongside the benchmark to bootstrap community contribution.
Evaluation Harness: Three Python-origin methods (Agentless, SWE-agent, OpenHands) are adapted into MagentLess, MSWE-agent, and MopenHands through prompt revision, context truncation, artifact filtering, and language-specific bug fixes. These methods are evaluated across nine LLMs on the 1,632-instance benchmark, with resolved rate as the primary metric.

Information flows: raw GitHub repositories → star/fork filtering → candidate repositories → PR crawling and issue-linking → Docker environment construction → automated test validation → human annotation → verified benchmark instances → adapted agent methods → LLM inference → patch generation → test execution → resolved rate measurement.

3.3 Roadmap for the Deep Dive

First, the overall benchmark construction philosophy: how the five-phase pipeline ensures diversity, executability, and human-verified correctness, and why each phase exists.
Second, Phase 1 (Repository Selection)—the criteria for choosing repositories and why popularity and CI/CD support are non-negotiable requirements.
Third, Phase 2 (PR Crawling)—how issue-linked, test-modifying, merged PRs are identified and what metadata is extracted.
Fourth, Phase 3 (Environment Determination)—the detailed process of building Docker images per-PR, classifying dependencies as repo-common vs. PR-specific, and the iterative debugging loop.
Fifth, Phase 4 (PR Filtering)—the three-configuration test execution framework (Run.log, Test.log, Fix.log) and the filtering rules that isolate genuine bug-fixing PRs, including the tracking of NONE and SKIPPED test statuses absent from SWE-bench.
Sixth, Phase 5 (Manual Verification)—the annotation protocol, annotator qualification, dual-annotation workflow, quality assessment, and the filtering rubric derived from SWE-bench Verified.
Seventh, the three adapted methods (MagentLess, MSWE-agent, MopenHands)—exactly what modifications were made to support multilingual evaluation and why each modification was necessary.
Eighth, the evaluation metrics and the difficulty categorization scheme based on estimated human resolution time rather than patch size metrics.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and empirical analysis paper whose core idea is that evaluating LLM-based issue resolving across multiple programming languages reveals fundamental capability limitations invisible in Python-only benchmarks.

Overall Construction Philosophy

The benchmark is built through a five-phase pipeline (Figure 2) that progressively transforms raw GitHub repositories into verified evaluation instances. The design philosophy has three pillars: diversity (seven languages, 39 repositories spanning 6.7k to 698.6k lines of code, multiple domains including web development, systems programming, and enterprise backends), executability (every instance runs in an isolated Docker container with all dependencies provisioned, ensuring reproducible evaluation), and human-verified correctness (every instance passes a dual-annotation quality check aligned with SWE-bench Verified standards). The authors explicitly note that Phases 1–4 are automated but Phase 5 requires human judgment, which is why the automated pipeline alone produces 4,723 instances (Multi-SWE-RL) while the verified subset is 1,632 instances (Multi-SWE-bench). This two-tier design is a pragmatic response to the cost of human verification: automated instances are sufficient for RL training where reward hacking risks are lower, while verified instances provide trustworthy evaluation.

Phase 1: Repository Selection

The goal of Phase 1 is to identify a set of GitHub repositories that are representative of real-world, production-level codebases in each of the seven target languages. The selection criteria are:

Popularity and Maintenance: repositories must have over 500 GitHub stars and demonstrate active maintenance for at least six months. The authors augment automated star filtering with manual prioritization of repositories "frequently recommended in Google searches using keywords such as 'high-quality', 'well-maintained', and 'popular'." This hybrid approach addresses a known limitation of star-count filtering: some widely-used infrastructure libraries (e.g., JSON parsers, logging frameworks) may have modest star counts relative to their impact, while some high-star repositories may be abandoned or poorly structured.

CI/CD Support: selected repositories must include CI/CD configurations (specifically, workflows under .github/workflows/). This is a structural requirement rather than a quality heuristic: automated testing infrastructure is necessary for Phase 3 (environment determination) and Phase 4 (PR filtering), where test suites must be executable in containerized environments. Without CI/CD configurations, extracting the correct build commands, dependency versions, and test invocation patterns becomes substantially more manual and error-prone.

Build Viability: the latest commit must be buildable and testable in a clean environment after minimal manual setup. This criterion eliminates repositories with complex, undocumented, or platform-specific build processes that would break the automated Docker construction in Phase 3. The phrase "minimal manual setup" is important: some level of human intervention is acceptable (e.g., specifying a JDK version or a compiler flag), but repositories requiring multi-hour environment debugging are excluded for scalability reasons.

Why these criteria? The paper could have selected repositories by language popularity rankings alone, but that strategy would optimize for representativeness of language usage rather than suitability for automated evaluation. The CI/CD and build viability requirements ensure that every selected repository can produce executable Docker environments—a practical constraint that makes the benchmark construction tractable at the cost of potentially excluding some important language ecosystems (e.g., embedded C codebases that require hardware-specific toolchains). The star-count threshold (>500) balances popularity (ensuring issues are meaningful to real users) with coverage (ensuring enough repositories exist per language).

The output of Phase 1 is explicitly stated as 39 repositories across 7 languages (Table 1), though the paper does not report how many repositories entered the filtering process. Notable inclusions: mui/material-ui (TypeScript, 27,632 files, 698.6k LoC—the largest repository in the benchmark), cli/cli (Go, 737 files, 165.1k LoC, contributing 397 instances—the most of any single repository), and sveltejs/svelte (JavaScript, 2,800 files, 105.9k LoC, 272 instances).

Phase 2: Pull Request Crawling

With repositories selected, Phase 2 identifies which pull requests can serve as issue-resolving instances. The process works in two steps:

Step 1: Collect all PRs from the repository. The paper does not specify the crawling mechanism (GitHub API, git log mining, or archive access), but the output is the complete PR history for each repository.

Step 2: Filter by three criteria:

Linked with at least one GitHub issue. The PR must reference an issue (e.g., via Fixes #1234 or manual linking in the GitHub UI). This criterion ensures the PR addresses a clearly defined task—a bug report or feature request with a natural language description that an LLM can use as input. PRs without linked issues (e.g., direct commits, refactoring PRs without corresponding issues) lack the "issue description → fix" structure that defines the benchmark task.
Modified test files. The PR must include changes to test files. This is a necessary condition for Phase 4 validation: if a PR doesn't modify tests, there are no test cases to verify the correctness of the fix. The paper does not specify whether "test files" are identified by naming convention (e.g., *Test.java, *_test.go), directory structure (e.g., src/test/), or manual inspection.
Merged into the main branch. The PR must be accepted by maintainers and integrated into the codebase. This serves as a weak signal of correctness—merged PRs have passed human code review and CI checks—and ensures that the fix patch represents a canonical solution rather than an abandoned attempt.

Metadata extraction. For each qualifying PR, the system extracts: issue description (the natural language text the LLM receives as input), base commit (the repository state before the fix, used as the starting point for LLM-generated patches), fix.patch (the ground-truth diff that resolves the issue), and test.patch (the diff that adds or modifies test cases associated with the fix). These four data elements—issue text, base commit SHA, fix diff, test diff—constitute the raw instance.

Why these criteria? The alternative would be to crawl all PRs and filter later, but the issue-linking and test-modification criteria serve as early pruning steps. A PR without an issue description cannot serve as a benchmark instance (no input to the LLM). A PR without test modifications cannot be automatically validated (no correctness signal). A PR that wasn't merged may represent an incorrect or incomplete fix. Together, these criteria reduce the pool to PRs that could plausibly become benchmark instances, saving substantial effort in Phases 3–4.

Phase 3: Environment Determination

This phase addresses a core challenge of repository-level evaluation: reproducibility. An LLM's generated patch must be applied to a specific repository state and tested in an environment identical to the one where the ground-truth fix was developed and tested. Phase 3 constructs Docker containers that provide this reproducibility.

The process is iterative and error-driven:

Step 1: Manual inspection of environment artifacts. For each PR, human engineers examine CI/CD configuration files (e.g., GitHub Actions YAML files), repository documentation (README files, CONTRIBUTING guides), and exploratory trial runs. The goal is to identify all software dependencies—compilers, interpreters, libraries, build tools, testing frameworks, and system packages—required to build and test the repository at the PR's commit.

Step 2: Dependency classification. Extracted dependencies are partitioned into two categories:

Repo-common dependencies: shared across the entire repository, independent of the specific PR. Examples: the Java Development Kit version for a Java project, the Go toolchain for a Go project, the npm or yarn package manager for a JavaScript project.
PR-specific dependencies: introduced or modified by the target PR. Examples: a new library dependency added as part of a feature implementation, a version bump of a testing framework.

This classification is manual and judgment-based. The paper provides no quantitative breakdown of how many dependencies are repo-common versus PR-specific, but the distinction matters for Dockerfile generation: repo-common dependencies form the base image layer (shared across all PRs from the same repository), while PR-specific dependencies are installed in a subsequent layer.

Step 3: Dockerfile generation and image building. A Dockerfile is automatically generated from the extracted dependency information. The Docker image is then built. If the build fails, the error logs are examined to identify missing dependencies, misconfigurations, or version conflicts.

Step 4: Iterative debugging. For fixable errors (missing packages, incorrect version pins, path issues), the Dockerfile or supporting scripts are patched iteratively until the image builds successfully. For unfixable errors (deep compatibility issues, unavailable dependencies, architectural incompatibilities), the PR is discarded. The paper does not report the discard rate at this stage, nor the distribution of common failure modes.

Step 5: Launch verification. Even after a successful Docker image build, the repository must actually launch and reach a functional state at the specific commit associated with the PR. The paper verifies that "all required services, packages, and configurations are functional." If launch fails, corrective actions are attempted; if successful, the containerized environment is validated for downstream use.

Why this approach? The alternative would be to use a pre-built container image per repository (e.g., the official SWE-bench Docker images). However, for multilingual support, such images don't exist. Each language ecosystem has its own build system, package manager, and dependency resolution conventions. A single Dockerfile template cannot cover Java (Maven/Gradle), JavaScript (npm/yarn/pnpm), Go (go modules), Rust (Cargo), and C/C++ (Make/CMake/autotools). The per-PR approach is more labor-intensive but guarantees that each instance's environment matches the exact dependency state at the time of the fix, which is critical for test reproducibility.

The paper notes that this phase "ensures that each PR is equipped with a clean and functional containerized environment, laying a necessary foundation for subsequent testing and analysis." The language reflects the engineering reality: environment determination is the largest source of manual effort in the pipeline, and its quality directly determines whether Phase 4's automated validation produces meaningful results.

Phase 4: Pull Request Filtering

With executable environments established, Phase 4 performs semantic validation to determine whether each PR genuinely represents a valid issue-resolving instance—one where a clear bug exists, is detectable by tests, and is verifiably fixed by the PR's patch.

The core mechanism is a three-configuration test execution framework:

For each PR, the full test suite is executed under three conditions:

Run.log: Tests executed on the base commit (the repository state before any changes from the PR). This establishes baseline test behavior.
Test.log: The test.patch is applied to the base commit before test execution. This simulates a state where the test cases that should catch the bug have been added, but the fix has not yet been applied. Ideally, the newly added tests (and possibly existing tests) should fail, revealing the bug.
Fix.log: Both test.patch and fix.patch are applied to the base commit before test execution. This simulates the fully fixed state. The failing tests from Test.log should now pass, and no previously passing tests should break.

A critical extension beyond SWE-bench: tracking four test statuses. While SWE-bench tracks only PASSED and FAILED outcomes, Multi-SWE-bench additionally tracks NONE and SKIPPED:

"Unlike SWE-bench, which considers only PASSED and FAILED, we also track NONE and SKIPPED status, as some test cases may be conditionally disabled or omitted after applying patches—resulting in inconsistent test counts across the three logs."

This matters because applying patches can change the set of test cases that exist or are executable. A test might be conditionally compiled (e.g., #ifdef in C/C++), dynamically skipped based on runtime configuration, or omitted entirely if a patch deletes or refactors test infrastructure. Without tracking NONE and SKIPPED, the filtering logic would misinterpret these cases as missing data.

The transition-based filtering logic. Each test case is summarized by its status transition across the three logs, represented as RunStatus → TestStatus → FixStatus. For example, a test that passes on the base commit, fails after applying test.patch, and passes after applying fix.patch is represented as PASSED → FAILED → PASSED—the canonical pattern for a correctly detected and fixed bug.

The filtering rules are:

Discard PRs with any ANY → PASSED → FAILED transitions. This means: if a test passes on the base commit, passes after applying test.patch, and then fails after applying fix.patch, the fix introduces a regression. Such PRs are excluded.
Discard PRs without at least one ANY → FAILED → PASSED transition. This means: there must be at least one test that fails after test.patch is applied and then passes after fix.patch is applied. Without this pattern, there is no evidence that the PR actually fixes a detectable bug.
Discard PRs with abnormal transitions such as PASSED → NONE/SKIPPED → FAILED. These patterns indicate ambiguous test behavior—a test that disappears and then reappears as failing cannot be reliably interpreted as a bug fix.

Output and test extraction. After applying these criteria, 2,456 instances are retained from the raw PR pool. For each retained instance, the system extracts test cases exhibiting transitions of the form Any → FAILED/PASSED/SKIPPED/NONE → PASSED and includes them in the dataset. These are the tests that the LLM's generated patch must pass to be considered correct.

Why this design? The three-configuration framework isolates the causal effect of fix.patch. By comparing Run.log (no changes) to Test.log (test patch only) to Fix.log (both patches), the system can distinguish between:

Tests that were already failing before the PR (not the PR's responsibility to fix)
Tests that fail due to the introduced test cases (the bug the PR should fix)
Tests that pass only after the fix is applied (evidence of correct resolution)
Tests that break due to the fix (regressions)

The ANY → FAILED → PASSED requirement is the key signal: it identifies tests that detect the bug (fail when only test infrastructure is added) and verify the fix (pass when the fix is applied). The ANY → PASSED → FAILED exclusion prevents instances where the fix breaks previously working functionality from entering the benchmark.

The inclusion of NONE and SKIPPED statuses reflects a practical reality of real-world codebases that simpler benchmarks can ignore: test suites are not static. Patches can change which tests exist, how they're configured, and whether they execute. A filtering framework that ignores these dynamics would produce false positives (treating a test that was deleted as "passing") or false negatives (discarding valid instances because test counts don't match).

Phase 5: Manual Verification

The final phase filters the 2,456 automatically validated candidates to 1,632 verified instances through human annotation. This phase is necessary because Phases 1–4 can identify PRs with clear test transitions but cannot assess whether the issue description is coherent, the unit tests actually cover the described bug, or the fix is appropriate—all factors that SWE-bench Verified identified as critical for evaluation validity.

Annotator recruitment and qualification. The paper recruits 68 annotators through outsourcing, with the number per language proportional to the remaining annotation workload. Qualification requirements: at least two years of experience in the target programming language and a relevant bachelor's degree or higher. These requirements are modest compared to expert annotation (which might require senior engineers), reflecting the practical tradeoff between annotation quality and the cost of annotating 2,456 instances across seven languages.

Training and support infrastructure. Before annotation, each annotator undergoes a one-hour training session covering:

Background on the issue-resolving task and the SWE-bench evaluation paradigm
Objectives of the Multi-SWE-bench project
Annotation procedures (what to look for, how to use the questionnaire)
Deliverables (completed annotation forms per instance)
Quality standards (alignment with SWE-bench Verified criteria)

Dedicated discussion channels are established for real-time guidance during annotation, enabling annotators to collaboratively resolve edge cases. This is a practical quality-control mechanism: in a multilingual benchmark, language-specific nuances (e.g., whether a particular test pattern in Rust adequately covers the described bug) may require discussion beyond the initial training.

Dual annotation with cross-review. Each instance is independently labeled by two annotators. Upon completion, the two annotations are cross-reviewed to produce a single, agreed-upon final label. This process is more rigorous than majority-vote aggregation: the cross-review forces annotators to reconcile disagreements, which both improves label quality and surfaces ambiguous instances.

Internal quality assessment. A team of 14 experienced engineers produces reference answers and verifies that the outsourced annotations for each language reach a minimum accuracy threshold of 80%. This is a cost-sensitive quality check: the internal team annotates only a subset of instances (enough to estimate accuracy), and if the outsourced annotations fall below 80% accuracy for a language, corrective action is taken (the paper does not specify whether this occurred or what corrective actions would entail).

The filtering questionnaire. The paper references a verification questionnaire (available at the provided GitHub URL) with three key questions:

Q2.1: "Serious Issue Flag" (Score 0 or 1). Score 0 appears to indicate the instance has a serious issue (e.g., incoherent description, missing test coverage, environment broken). The filtering criterion requires Q2.1 = 0, meaning instances with serious issues are excluded.
Q3.1: "Clarity of Issue Description" (Score 0–3, with 3 being most clear). The filtering criterion requires Q3.1 ∈ {2, 3}, meaning only instances with sufficiently clear issue descriptions are retained. Score 0–1 instances have descriptions that are ambiguous, incomplete, or misleading—LLMs would lack the necessary information to produce a correct fix regardless of capability.
Q4.1: "Coverage of Unit Tests" (Score 0–3, with 3 being most comprehensive). The filtering criterion requires Q4.1 ∈ {2, 3}, meaning only instances where the unit tests adequately cover the described issue are retained. Score 0–1 instances have test suites that don't actually verify the fix, making automated evaluation unreliable.

Annotation outcomes. Table 3 provides the scoring statistics per language. Key observations:

Q2.1: Most instances show Score 0 (no serious issue). Java has 146 Score 0 vs. 10 Score 1; TypeScript has 382 Score 0 vs. 8 Score 1. The low rate of serious issue flags validates the automated filtering in Phases 1–4.
Q3.1: Scores are concentrated in the upper range (2–3). JavaScript has 567 instances with Score 3 (out of 590 total). Go has 288 Score 3 and 276 Score 2. C has 79 Score 3 and 115 Score 2. This suggests issue descriptions in most repositories are sufficiently clear.
Q4.1: Scores are more distributed. TypeScript shows 142 Score 3 vs. 133 Score 2 vs. 76 Score 1, indicating more variability in test coverage adequacy. JavaScript has only 54 Score 3 out of 590 instances, with 305 Score 2 and 172 Score 1—suggesting test coverage is a more significant concern for JavaScript repositories.

Why this design? The manual verification phase is the paper's primary quality-control investment. Without it, Multi-SWE-bench would inherit the issues that motivated SWE-bench Verified: automated filtering can identify PRs with valid test transitions but cannot assess whether those tests actually test the described bug or whether the issue description contains sufficient information for a human (or LLM) to diagnose and fix the problem. By adopting SWE-bench Verified's annotation standards, the paper ensures that Multi-SWE-bench instances meet the same quality bar as the de facto standard Python benchmark.

The 80% accuracy threshold for outsourced annotations is a pragmatic compromise: perfect annotation accuracy would require exclusively using senior engineers (prohibitively expensive at this scale), while no quality threshold would risk data contamination. The threshold is applied per-language, reflecting the reality that annotation difficulty varies by language (e.g., TypeScript may be harder to annotate than Go due to dynamic typing and asynchronous execution patterns).

Difficulty Categorization: Time-Based Rather Than Metric-Based

Unlike SWE-bench, which does not assign difficulty levels, Multi-SWE-bench introduces a difficulty categorization based on estimated human resolution time, recorded during manual annotation. Issues are categorized into three levels:

Easy: ≤15 minutes of estimated human effort
Medium: 15 minutes to 1 hour
Hard: ≥1 hour

The distribution across languages (Table 2, Figure 3) reveals substantial variation. TypeScript has 72 easy, 88 medium, and 64 hard instances. JavaScript has only 10 easy instances but 241 hard ones. Go has 141 easy, 153 medium, and 134 hard. Rust has 66 easy, 126 medium, and 47 hard.

Why time-based rather than metric-based? The paper argues that superficially similar metrics like token count or file span can be misleading:

"certain easy instances exhibit large-scale edits (e.g., Rust), which are typically due to highly repetitive and pattern-consistent changes. This highlights the advantage of time-based difficulty categorization over superficial metrics like token count or file span."

For example, a Rust instance requiring 1,600 lines of changes might be categorized as "easy" if those changes are mechanical refactoring (e.g., renaming a type across 50 files following a consistent pattern), while a 10-line JavaScript change might be "hard" if it requires reasoning about asynchronous execution order across multiple event handlers. The time-based annotation captures the cognitive complexity of the fix rather than its textual extent.

Table 2 validates this categorization by showing that, as difficulty increases, fix patches generally involve more lines, hunks, and files—but the relationship is not deterministic. Easy Rust instances average 318.7 lines of fix patches versus 5.0 lines for easy Python, yet both are categorized as easy because the Rust changes are pattern-consistent rather than conceptually complex.

Why this matters for evaluation. The difficulty categorization enables fine-grained analysis (Section 6.1.1, Table 5) that reveals a stark finding: across all methods and languages, hard issues (>1 hour) show near-zero resolved rates. This finding—that current LLMs can only handle issues solvable by humans in under 15 minutes—would be invisible without difficulty annotation, because aggregate resolved rates would obscure the complete failure on complex tasks.

The Multi-SWE-RL Dataset: Automated Breadth for RL Training

Alongside the manually verified Multi-SWE-bench (1,632 instances), the paper releases Multi-SWE-RL: 4,723 containerized issue-resolving instances spanning 76 repositories and seven languages, produced by the same pipeline (Phases 1–4) "excluding the manual verification process described in Sec. 3.1.5."

The 4,723 count versus the 2,456 retained after Phase 4 requires explanation. The paper states that Multi-SWE-RL instances come from 76 repositories, whereas the 2,456 Phase 4 candidates came from 39 repositories (Table 1). This suggests that the Multi-SWE-RL dataset was produced by running Phases 1–4 on additional repositories beyond the 39 selected for Multi-SWE-bench, or that the Phase 4 filtering was less stringent for the RL dataset. This ambiguity is not resolved in the paper.

Design rationale. Manual verification (Phase 5) is the bottleneck: it requires 68 annotators and takes substantial time. For RL training, where reward signals come from test execution rather than human evaluation, the risk of data quality issues is lower. An instance with an unclear issue description might produce noisy training signals, but the RL agent can learn to handle ambiguity or simply fail on that instance. An instance with incomplete test coverage might give false positive rewards, but these can be detected by evaluating on the verified benchmark. The two-tier design—verified instances for evaluation, unverified instances for training—is a pragmatic response to the cost of human annotation at scale.

Community infrastructure. The Multi-SWE-RL release includes the complete data construction pipeline (reproduction scripts, Dockerfile templates, dependency extraction tools) and detailed contribution tutorials. The paper explicitly invites community members to contribute new instances for additional repositories and languages, with a contribution incentive plan that includes co-authorship on quarterly arXiv updates and leaderboard recognition.

Method Adaptation: From Python-Specific to Multilingual

The paper evaluates three methods originally designed for Python on SWE-bench. To enable multilingual evaluation, each method required modifications. These adaptations are described in Section 5.1.

MagentLess (adapted from Agentless):

Agentless (Xia et al., 2024) is a fixed-workflow approach that resolves issues through a multi-stage pipeline: hierarchical fault localization (identifying which files and functions are relevant to the issue), code repair (generating candidate patches for the identified locations), and candidate patch selection via regression and reproduction tests. MagentLess makes five modifications:

Prompt revision for multilingual support. All prompts (for fault localization, code repair, and patch validation) are rewritten to accommodate the newly added languages. The paper does not provide the revised prompts but notes that they differ from the Python-originals because language-specific concepts (e.g., "class" vs. "struct" vs. "trait", "import" vs. "use" vs. "require") must be reflected in the instructions.
Full file content replaces file skeletons. In Agentless, file skeletons (truncated views showing function signatures but not implementations) are used to reduce context length. This approach fails for languages where extracting skeletons is "challenging"—the paper cites no specific examples, but likely candidates include C/C++ preprocessor macros that obscure function boundaries, TypeScript decorators that modify class structure, and Rust macros that generate code not present in the source. By using full file content, MagentLess avoids parse failures at the cost of increased token consumption. Table 7 confirms this: MagentLess on TypeScript averages 241,180 input tokens (GPT-4o), nearly 7× the Python average of 36,150 tokens.
Tree-sitter for cross-language code extraction. Agentless extracts file skeletons, classes, and functions to construct context windows around candidate edit locations. MagentLess replaces the Python-specific extraction with Tree-sitter, a parser generator tool with grammars for all seven target languages. This is a principled choice: Tree-sitter is language-agnostic, handles incomplete or syntactically malformed code gracefully, and is actively maintained. However, Section 6.3.2 reveals a limitation: Tree-sitter "fails to reliably extract code structures in JavaScript repositories that use loosely bound syntax such as arrow functions," preventing MagentLess from constructing context windows for certain JavaScript instances.
Repository structure pruning by file extension. TypeScript repositories (particularly mui/material-ui with 27,632 files) can produce repository structure listings that exceed LLM context limits. MagentLess prunes the extracted structure by retaining only files and directories with specific extensions (e.g., .ts, .tsx). This is a crude but effective fix: it reduces context length at the risk of excluding relevant non-TypeScript files (e.g., configuration files, documentation, build scripts).
Removal of candidate patch selection stage. Agentless uses regression and reproduction tests to select among candidate patches. MagentLess removes this stage entirely and retains only fault localization and code repair, because "regression and reproduction testing is cumbersome to implement across languages and falls outside the scope of this work." This is a significant capability reduction: Agentless's selection stage is important for filtering incorrect patches, and its removal likely reduces MagentLess's resolved rate compared to what a fully language-adapted version could achieve.

MSWE-agent (adapted from SWE-agent):

SWE-agent (Yang et al., 2024) is an agent-based approach where the LLM interacts with the codebase through an Agent-Computer Interface (ACI)—a set of predefined commands (e.g., find, grep, edit, submit) that the agent invokes in multi-turn interactions. MSWE-agent makes four modifications:

Prompt revision for multilingual support. All prompts (the system prompt defining the ACI, the task description prompt, and any few-shot examples) are revised to accommodate non-Python languages.
Observation truncation. SWE-agent environments can produce extremely long observations (e.g., grep output for a common pattern in a large codebase, terminal output from a failing build). In Python, these can often be kept within context limits by restricting search scope. In other languages—particularly TypeScript and C++ with their large compilation outputs—observations consistently exceed limits. MSWE-agent "truncated overly long environment observations to ensure stable agent execution." The truncation strategy (head-only, head-and-tail, extractive summary) is not specified.
.gitignore for compiled artifacts. In C and C++, compilation produces binary artifacts (.o files, .bin executables, .so shared libraries) that git apply cannot handle if they appear as untracked files in the working directory. MSWE-agent adds .gitignore entries to exclude these artifacts. This is a language-specific workaround: Python repositories rarely produce compiled artifacts, so SWE-agent never needed this fix.
Language-specific command fixes. Certain commands "caused crashes or non-terminating behavior during execution." The paper does not enumerate these commands but implies they are language-specific (e.g., a Go build command that hangs on circular imports, a Rust cargo test invocation that requires specific feature flags).

MopenHands (adapted from OpenHands):

OpenHands (Wang et al., 2024b) is an interactive agent platform where the LLM uses actions (e.g., CmdRunAction, FileReadAction, FileWriteAction) to interact with the environment. MopenHands makes three modifications:

Prompt revision for multilingual support. All prompts are rewritten for the seven target languages.
.gitignore for compiled artifacts. Same as MSWE-agent: binary artifacts from C/C++ compilation interfere with git apply.
Fix for tab character rendering in CmdRunAction. This is a specific implementation bug: the CmdRunAction (which executes shell commands and captures output) "incorrectly rendered tab characters (\t) as spaces in git diff outputs, making patches unapplicable." The fix redirects diff output to a file and reads it using FileReadAction, which preserves tab characters. This bug was "especially important in languages like Go," where tabs are semantically significant (Go uses tabs for indentation in its standard formatting). This reveals a subtle but critical point: adapting Python-first tools to multilingual settings exposes assumptions hardcoded into the tools themselves—in this case, an assumption that whitespace rendering is cosmetic rather than semantic.

Why adapt existing methods rather than build new ones? The paper's choice to adapt rather than redesign reflects both practical constraints and scientific goals. Practically, building language-first agents for seven languages would be a massive engineering effort that the paper's authors (primarily a benchmark construction team) are not positioned to undertake. Scientifically, evaluating Python-first methods on non-Python languages directly tests the paper's central question: does Python performance generalize? If Python-first methods performed well on other languages after minimal adaptation, that would support the generalization hypothesis. The fact that they perform poorly (as Section 6 demonstrates) provides evidence for the opposite conclusion: the methods are Python-specialized, and substantial redesign is needed for multilingual capability.

Caveat: Adaptation quality is a confound. The paper acknowledges that the adaptations are incomplete:

"there remains substantial room for improvement, particularly in language-specific adaptation and overall robustness."

This matters because the observed cross-language performance drops (Section 6) could be partially attributable to suboptimal adaptation rather than genuine LLM limitations. For example, MagentLess's removal of the patch selection stage reduces its effectiveness compared to the full Agentless pipeline; MSWE-agent's observation truncation might discard critical information; MopenHands's prompt revisions might not adequately convey language-specific concepts. The paper treats these as part of the evaluation ecosystem—future work can improve the methods and re-evaluate—but the reader should understand that the reported resolved rates represent current capability with Python-first methods, not an upper bound on what language-optimized methods could achieve.

Evaluation Metrics

The primary evaluation metric is Resolved Rate (%): the percentage of issues for which the LLM's generated patch passes all extracted test cases. This is the same metric used by SWE-bench and SWE-Lancer.

Additional metrics include:

Success Location (%): the accuracy of fault localization at the file level—whether the LLM correctly identifies which files need modification. This is computed by comparing the files touched by the LLM's generated patch against the files touched by the ground-truth fix.patch. The metric uses only file-level granularity (not line-level), so an LLM that correctly identifies the file but edits the wrong lines would score 100% on Success Location but 0% on Resolved Rate.
**Average Cost ( $)**: the average cost per issue, computed from the LLM provider's token pricing for input and output tokens. This metric enables cost-effectiveness comparisons across models and methods. Table 8 reports these costs; for example, DeepSeek-V3 + MagentLess on Java averages$ 0.0059 per issue, while OpenAI-o1 + MSWE-agent on Python averages $3.75 per issue—a 635× difference.
Turn distribution (for MSWE-agent and MopenHands): the number of interaction turns required to successfully resolve an issue. Figure 5 shows this as box-and-whisker plots per language and model. The metric captures method efficiency: a method that resolves issues in fewer turns is more practical for deployment even if it achieves the same resolved rate.

Why multiple metrics? Resolved rate alone can mask important dynamics. A method with high resolved rate but poor fault localization accuracy (Success Location) may be guessing correctly by chance rather than genuinely understanding the issue. High resolved rate at extreme cost may be impractical. A method requiring 50 turns to resolve issues may be too slow for interactive use. The multi-metric evaluation provides a more complete picture of method capability.

4. Key Insights and Innovations

Innovation 1: Multilingual Issue Resolving as a Distinct Capability Frontier

The paper's most foundational contribution is not the benchmark itself but the demonstration that Python issue-resolving performance is not a valid proxy for general software engineering capability. Before Multi-SWE-bench, the field implicitly treated SWE-bench progress—from 0.40% to 65.40% resolved rate in under a year—as evidence that LLMs were approaching competence as autonomous software engineers. The unstated assumption was that Python, being a general-purpose language with a large open-source ecosystem, was representative enough that performance gains would transfer to other languages with modest adaptation.

Multi-SWE-bench decisively refutes this assumption. The evidence is unambiguous: Claude-3.7-Sonnet, the best-performing model across the evaluation, achieves 45.80% on Python (MSWE-agent) versus 23.44% on Java, 11.16% on TypeScript, 4.78% on JavaScript, 5.37% on Go, 6.69% on Rust, 8.59% on C, and 11.63% on C++ (Table 4). The gap is not uniform—it ranges from roughly 2× (Java) to nearly 10× (JavaScript)—but it is consistent across all models and methods. Even allowing for suboptimal method adaptation (discussed in Section 3), a 2× to 10× performance cliff cannot be explained by prompt engineering or observation truncation alone.

What makes this a conceptual shift rather than just a measurement? The finding redefines what "progress on software engineering" means. Prior to this work, a paper reporting a 5% improvement on SWE-bench could claim to have advanced "automated software engineering." After Multi-SWE-bench, that claim requires qualification: the advance may be Python-specific until demonstrated otherwise. This is analogous to the shift in NLP when researchers realized that English-only benchmarks (GLUE, SuperGLUE) did not measure general language understanding—they measured English language understanding, and multilingual benchmarks (XTREME, XGLUE) revealed capability gaps invisible in monolingual evaluation. Multi-SWE-bench does for code what those benchmarks did for natural language: it exposes that the evaluation surface was systematically biased toward one language ecosystem, and that the bias was large enough to produce a misleading picture of overall capability.

The significance extends beyond benchmarking. It implies that the techniques developed for Python issue resolving—hierarchical fault localization, agent-computer interfaces, repository structure extraction—may embed Python-specific assumptions that don't transfer. For example, Agentless's file skeleton extraction (removed in MagentLess for multilingual support) relies on Python's relatively simple syntax for function and class boundaries, an assumption that fails for C preprocessor macros, Rust procedural macros, or TypeScript decorators. The agent loop design in SWE-agent assumes that repository exploration commands (find, grep, ls) produce manageable output, an assumption violated by TypeScript repositories with 27,000+ files. These aren't implementation details; they're architectural choices that reflect implicit Python-centrism in the entire agent design paradigm.

Innovation 2: Time-Based Difficulty as a Cross-Language Normalization Strategy

The paper introduces a difficulty categorization for issue-resolving instances based on estimated human resolution time (≤15 minutes, 15 minutes–1 hour, ≥1 hour), recorded during the manual annotation phase. This is a genuinely novel contribution to benchmark design in software engineering, and its intellectual significance lies in solving a problem that metric-based difficulty measures create for multilingual evaluation.

The problem with metric-based difficulty. Before Multi-SWE-bench, researchers analyzing SWE-bench results sometimes used proxy metrics for difficulty: lines of code changed in the fix patch, number of files modified, token length of the issue description, or repository size. These metrics have face validity—larger changes seem harder—but they are not language-invariant. A 1,600-line Rust fix might be a mechanical rename across 50 files (easy), while a 10-line JavaScript fix involving asynchronous callback ordering might require deep reasoning about event loop semantics (hard). If difficulty is defined by patch size, these two instances would be misclassified relative to their actual cognitive demands, and any analysis of "model performance vs. difficulty" would be contaminated by language-specific correlations between patch size and genuine complexity.

How time-based annotation solves this. By asking human annotators to estimate how long a competent developer would need to resolve the issue, the paper anchors difficulty to a cross-language invariant: human cognitive effort. A 15-minute bug is a 15-minute bug regardless of whether the fix requires 3 lines of Java or 300 lines of mechanically-generated Rust. This normalization is what enables the paper's per-difficulty analysis (Table 5) to be meaningful across languages. Without it, comparing resolved rates on "hard" C++ instances versus "hard" JavaScript instances would be comparing apples to oranges—the C++ instances might be "hard" because they involve 800-line patches while the JavaScript instances might be "hard" because they involve subtle async logic, and we'd have no way to know whether the performance difference reflects language difficulty or metric artifact.

The validation of this approach is implicit in Table 2: as difficulty increases, fix patches do tend to involve more lines, hunks, and files—confirming that time correlates with textual extent—but the relationship is not deterministic. Easy Rust instances average 318.7 lines of fix patches (versus 5.0 lines for easy Python), exactly the kind of cross-language discrepancy that would confuse metric-based difficulty categorization. The time-based approach correctly classifies these as easy because the changes are pattern-consistent rather than conceptually demanding.

Why this matters beyond this paper. The time-based difficulty concept is portable to any software engineering benchmark where the task complexity cannot be reduced to surface features of the ground-truth solution. Code generation benchmarks (HumanEval, MBPP) could annotate problem difficulty by estimated human solve time rather than relying on test-case count or solution length. Repository-level benchmarks across other domains (code translation, refactoring, documentation generation) could adopt the same approach. It is a small methodological innovation—one additional field in the annotation questionnaire—but it enables analyses that would otherwise be confounded by cross-instance heterogeneity in what "difficulty" means.

Innovation 3: The Issue Flow Analysis as a Diagnostic Framework

Section 6.1.2 introduces what the paper calls the "issue flow" (Figure 4): a Sankey-style visualization tracking how issues progress through the resolution pipeline, from submission to fault localization success/failure to final resolution success/failure. This is not merely a visualization choice—it is a diagnostic decomposition that separates the two fundamental sub-tasks of issue resolving (locating the relevant code and editing it correctly) and reveals where different methods fail.

The diagnostic insight. Figure 4 shows that for all three methods, the majority of failures occur at the fault localization stage rather than the code editing stage. For example, with Claude-3.7-Sonnet + MagentLess on Java, the flow shows that many issues are never successfully located—the agent cannot identify which files need modification. Among those that are located, a higher proportion are resolved. This pattern is consistent across languages and suggests that improving fault localization is the highest-leverage intervention for boosting resolved rates—a non-obvious finding given that the literature often focuses on code generation quality (e.g., better patch ranking, more sophisticated repair strategies).

More subtly, Figure 4 reveals an asymmetry between MagentLess and the two agent-based methods. MagentLess achieves higher fault localization accuracy than MSWE-agent and MopenHands but lower overall resolved rates. This means MagentLess is better at finding the right files but worse at producing correct fixes once it finds them—a direct consequence of MagentLess's removal of the candidate patch selection stage (noted in Section 3.4) and its rigid workflow that limits the repair step's flexibility. The agent-based methods are worse at localization (they get lost exploring large repositories) but better at repair (their interactive loops allow iterative refinement). This diagnostic decomposition would be invisible in aggregate resolved rate comparisons, which would simply show that MopenHands outperforms MagentLess without explaining why.

Comparison to prior diagnostic approaches. SWE-bench papers typically report resolved rate and sometimes provide coarse failure categorization (e.g., "patch failed to apply," "tests failed"), but they do not systematically decompose the resolution process into location and editing stages. SWE-agent's original paper analyzed which ACI commands were most frequently used but did not separate localization accuracy from repair accuracy. The issue flow is a lightweight diagnostic—it requires only comparing the files touched by the generated patch against ground-truth files—but it surfaces structural information about method capabilities that aggregate metrics conceal.

Significance for method design. The diagnostic points toward a hybrid architecture: use MagentLess-style hierarchical localization (which is accurate) to identify candidate files, then hand off to an interactive agent (MSWE-agent or MopenHands) for iterative repair within those files. The paper does not propose this architecture, but the issue flow analysis makes the case for it empirically. Future methods can use the same diagnostic to evaluate whether their improvements come from better localization or better repair, enabling more targeted iteration.

Innovation 4: The Multi-SWE-RL Community as an Infrastructure Strategy

The paper's dual release of Multi-SWE-bench (evaluation) and Multi-SWE-RL (training data + community infrastructure) represents a strategic innovation in benchmark-driven research that goes beyond the typical "release dataset, publish leaderboard" model.

What's distinctive about this approach. Benchmark papers almost universally treat the dataset as the final deliverable, with future extensions left to the authors or unaddressed entirely. Multi-SWE-bench explicitly rejects this static model. The paper announces a rolling update schedule (quarterly arXiv revisions), a contribution incentive plan (co-authorship for contributed instances and RL results), a two-tier data quality architecture (automated instances for training via Phases 1–4, manually verified instances for evaluation via Phase 5), and detailed contribution tutorials. This transforms the benchmark from a product into a platform—a living research infrastructure that the community extends rather than a fixed snapshot that the community consumes.

Why this matters now. The paper argues—correctly—that "the creation of such interactive environments and data trajectories is extremely challenging," noting that Multi-SWE-bench's 1,632 verified instances took approximately one year to produce. No single research group can build comprehensive multilingual software engineering benchmarks at the scale needed for RL training (which may require tens of thousands of instances across dozens of languages). The community contribution model is not merely aspirational; it's a practical necessity for the research agenda the paper advocates (scaling RL in real-world software environments).

The incentive structure is also notable. By offering co-authorship on quarterly updates for contributed instances, models, and results, the paper creates a currency (authorship) that aligns community incentives with benchmark growth. This borrows from the open-source software community's governance model (contribution → recognition → maintainer status) rather than the typical academic benchmark model (citation → use → eventual obsolescence). Whether this incentive structure will work at scale is unknown—the paper is too new to have attracted community contributions yet—but the design itself is a contribution to the meta-problem of how to build and maintain large-scale evaluation infrastructure in AI research.

Relationship to the pretraining-inference scaling narrative. The multi-SWE-RL community is positioned within a specific research bet: that reinforcement learning with verifiable reward signals (test pass/fail) will produce substantial improvements in software engineering capability, following the trajectory of DeepSeek-R1 and OpenAI-o1 in reasoning domains. If this bet pays off, the bottleneck shifts from "can we train capable agents?" to "do we have enough training environments across enough languages to prevent overfitting?" Multi-SWE-RL is infrastructure against this future bottleneck, built before the bottleneck arrives. This is a strategically timed contribution: by releasing training data and community tooling simultaneously with the evaluation benchmark, the paper aims to catalyze the RL research it argues is necessary, rather than waiting for the field to independently recognize the need and build the infrastructure from scratch.

5. Experimental Analysis

Evaluation Methodology

Dataset. Multi-SWE-bench consists of 1,632 human-validated issue-resolving instances spanning 7 programming languages (Java, TypeScript, JavaScript, Go, Rust, C, C++) sourced from 39 GitHub repositories, plus Python baselines from SWE-bench. The dataset is split by language and further stratified into three difficulty levels (easy, medium, hard) based on estimated human resolution time recorded during the manual annotation phase (Phase 5).
Base model(s). Nine frontier LLMs are evaluated: GPT-4o (gpt-4o-2024-11-20), OpenAI-o1 (o1-2024-12-17), OpenAI-o3-mini-high (o3-mini-2025-01-31 high reasoning effort), Claude-3.5-Sonnet (claude-3-5-sonnet-20241022), Claude-3.7-Sonnet (claude-3-7-sonnet-20250219), DeepSeek-V3, DeepSeek-R1, Qwen2.5-72B-Instruct, and Doubao-1.5-pro. These models span a range of architectures (dense, mixture-of-experts, reasoning-specialized), capabilities, and price points, chosen to provide broad coverage of the frontier model landscape. All models are evaluated through API access with provider-specific pricing used for cost calculations.
Metrics. The primary metric is Resolved Rate (%) — the percentage of instances for which the LLM-generated patch passes all extracted test cases associated with the ground-truth fix. Evaluation follows the SWE-bench protocol: the generated patch is applied to the base commit, the test suite is executed, and pass/fail is determined by comparing test outcomes against the expected transitions. Additional metrics include Success Location (%) (file-level fault localization accuracy, computed by checking whether the files touched by the generated patch intersect with the ground-truth fix patch files), Average Cost ($) (per-issue API cost computed from provider token pricing for input and output tokens), and for interactive methods, turn count distributions (the number of interaction rounds required for successful resolution, visualized as box plots in Figure 5).
Baselines. The evaluation does not use baselines in the traditional sense (there is no prior multilingual issue-resolving benchmark to compare against). Instead, three methods serve as evaluation harnesses that test the same LLMs under different interaction paradigms: MagentLess (adapted from Agentless [Xia et al., 2024]), a fixed multi-stage workflow with hierarchical fault localization followed by code repair; MSWE-agent (adapted from SWE-agent [Yang et al., 2024]), an agent-based approach using a predefined Agent-Computer Interface for multi-turn repository interaction; and MopenHands (adapted from OpenHands [Wang et al., 2024b]), an interactive agent platform where LLMs use actions (CmdRunAction, FileReadAction, FileWriteAction) to navigate and modify the codebase. Python performance on SWE-bench Verified is reported as a reference point for comparison across languages.
Generation budget / compute accounting. There is no uniform generation budget across methods because the three approaches consume compute differently. MagentLess operates in a fixed workflow with a predetermined number of LLM calls (fault localization pass + repair pass, without the patch selection stage). MSWE-agent and MopenHands operate in multi-turn loops capped at 50 interaction rounds per instance. Cost accounting uses actual API token consumption multiplied by provider prices (input and output tokens tracked separately) and is reported as average cost per issue in USD (Table 8). Token consumption is measured using the GPT-4o tokenizer for all models to enable cross-model comparison (Table 7), though this undercounts tokens for models using different tokenizers.
Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The evaluation is a single-pass measurement: each model–method pair is run once on each of the 1,632 instances (with some exceptions — the paper does not clarify whether every model was evaluated on every instance or whether partial evaluations occurred). The paper does not report confidence intervals, standard deviations, or any measure of variance in resolved rates. The only cross-validation-like procedure is the dual annotation with cross-review in Phase 5 manual verification, which applies to benchmark construction (ensuring instance quality) rather than to evaluation methodology (ensuring result reliability). This absence is a notable methodological gap: with 1,632 instances split across 7 languages, per-language sample sizes range from 128 (Java) to 590 (JavaScript), and per-difficulty-bin sample sizes can be as small as 10 (JavaScript easy instances), making variance potentially large.

Main Quantitative Results

Language-Level Performance: Python Dominance and the Generalization Cliff

The headline finding from Table 4: no model–method combination achieves a resolved rate above 23.44% on any non-Python language, despite Python rates exceeding 50%. Claude-3.7-Sonnet + MopenHands achieves 52.20% on Python but only 21.88% on Java, 2.23% on TypeScript, 5.06% on JavaScript, 7.48% on Go, 15.90% on Rust, 8.59% on C, and 14.73% on C++. The Python-to-Java gap (52.20% → 21.88%, a 2.4× drop) is the smallest; the Python-to-TypeScript gap (52.20% → 2.23%, a 23× drop) is the largest.

The performance hierarchy across language domains is consistent (Section 6.1.1):

"the performance generally follows a hierarchy, with high-level general-purpose languages outperforming systems programming and low-level/high-performance computing languages, while web development languages perform the worst."

Specifically, Java ranks second after Python across most model–method combinations, with resolved rates clustering in the 10–23% range depending on the model (Table 4, MagentLess column for Java: from 5.47% for Doubao-1.5-pro to 22.66% for DeepSeek-R1). Go and Rust occupy a middle tier, with Claude-3.7-Sonnet achieving 7.48% and 15.90% respectively via MopenHands, but other models often dropping below 5%. C and C++ show high variance: Claude-3.5-Sonnet + MopenHands reaches 12.40% on C++, while Doubao-1.5-pro + MagentLess scores 0.00% on both C and C++. TypeScript and JavaScript form the bottom tier, with most model–method combinations below 5% resolved rate.

Table 5 (difficulty-stratified results) reveals that this aggregate picture masks a critical threshold effect. On easy issues (≤15 minutes estimated human effort), performance is substantially higher: Claude-3.7-Sonnet + MSWE-agent achieves 44.44% on easy Java and 48.15% for Claude-3.5-Sonnet on the same method and difficulty. However, on hard issues (≥1 hour), resolved rates approach zero across all models, methods, and languages:

"For hard-level issues, existing LLMs and agents are mostly ineffective, with resolved rates approaching zero."

The paper provides no precise threshold — "approaching zero" is qualitative — but scanning Table 5's Hard columns confirms: hard Java peaks at 3.13% (OpenAI-o1 + MSWE-agent), hard TypeScript at 3.13% (OpenAI-o1 + MSWE-agent), hard Rust at 14.89% (Claude-3.7-Sonnet + MopenHands — a notable outlier), and hard C++ at 2.38% (DeepSeek-V3 + MSWE-agent). Most hard entries are 0.00%. The Rust outlier (14.89% hard resolved rate) is unexplained in the text but may relate to the phenomenon noted in the difficulty discussion: "certain easy instances exhibit large-scale edits (e.g., Rust), which are typically due to highly repetitive and pattern-consistent changes." Some "hard" Rust instances by time estimate may still be mechanically solvable if they involve pattern-consistent bulk edits.

Table 6 (issue type breakdown) adds further granularity for Claude-3.7-Sonnet. Bug fix issues consistently show the highest resolved rates, followed by new features, with feature optimization being the hardest:

"bug fix issues are resolved with the highest success rates, followed by new features, with feature optimization being the most challenging."

For example, MSWE-agent on Java: 17.97% Bug Fix, 3.91% New Feature, 1.56% Feature Optimization. MopenHands on Rust: 12.97% Bug Fix, 2.93% New Feature, 0.00% Feature Optimization. This hierarchy holds across all languages and methods.

Key caveat on Python comparison: The Python numbers in Tables 4–5 are not directly comparable to the other languages because they use the original SWE-agent and Agentless implementations (without the modifications applied to MSWE-agent and MagentLess) and are evaluated on SWE-bench Verified rather than Multi-SWE-bench instances. The paper clarifies this in Section 6.4 for MSWE-agent: "we maintain the original SWE-agent implementation for Python, which does not incorporate the over-length truncation mechanism applied to other languages." This means Python benefits from method implementations that have been optimized and iterated on by the original authors, while other languages use adaptations made by the Multi-SWE-bench team with "substantial room for improvement." The Python numbers should be understood as an upper reference point showing what methods can achieve in their native domain, not as a controlled comparison against Multi-SWE-bench languages.

Method-Level Performance: Interactive Agents Outperform Fixed Workflows

Comparing across methods in Table 4 and Table 5, MopenHands achieves the highest resolved rate in the majority of language–model combinations, while MagentLess generally trails. Specifically, Section 6.1.2 reports:

"Overall, MopenHands outperforms the others in most cases, achieving the highest resolved rate in five out of seven languages, while MSWE-agent wins twice and MagentLess wins once."

The wins are: MSWE-agent wins on Java (23.44% with Claude-3.7-Sonnet vs. 21.88% for MopenHands) and TypeScript (11.16% vs. 2.23%), while MagentLess wins on JavaScript (1.97% for Claude-3.7-Sonnet vs. 4.78% for MSWE-agent and 5.06% for MopenHands — this claim in the text appears inconsistent with Table 4 data and may reflect averaging across models rather than peak performance).

The paper attributes MopenHands's and MSWE-agent's advantage to "their more flexible workflow, which is better suited to another language beyond Python compared to MagentLess." MagentLess's rigid pipeline (fixed fault localization → repair) limits adaptability, and — critically — the removal of the candidate patch selection stage (one of the five MagentLess adaptations) eliminates a quality-control step that Agentless uses to filter incorrect patches.

However, an important exception is noted:

"a notable exception to this general trend is observed in the performance of the models DeepSeek-R1 and Qwen2.5-72B-Instruct. For these two models, MagentLess generally provides better results than MSWE-agent for languages except C and C++."

This suggests that reasoning-specialized models (DeepSeek-R1) and smaller open-weight models (Qwen2.5-72B-Instruct) may be less effective at the multi-turn agent loop, perhaps because their training or architecture is less suited to tool-use interactions, and benefit from the structured, single-pass workflow of MagentLess.

The Issue Flow: Localization is the Primary Bottleneck, Especially for Agent-Based Methods

Figure 4 presents the "issue flow" — tracking whether issues are successfully localized (the generated patch touches at least one ground-truth fix file) and subsequently resolved. The key diagnostic finding (Section 6.1.2):

"all three methods generally fail to locate issues more often than they succeed. Accurate issue localization is fundamental to the overall success of the resolution process, serving as a prerequisite for effective code editing."

The flow reveals a counterintuitive asymmetry for Claude-3.7-Sonnet: MagentLess achieves higher fault localization accuracy than MSWE-agent and MopenHands, but lower overall resolved rates. This means the fixed workflow is better at finding the right files (hierarchical search is systematic) but worse at producing correct edits once it finds them (the rigid repair step, without patch selection, generates lower-quality fixes). The interactive agents are worse at localization (they get lost or distracted in large repositories during multi-turn exploration) but better at repair (iterative refinement produces higher-quality patches).

The implication for future method design is explicit: "This underscores the need for a balanced method that not only prioritizes precise issue identification but also enhances the model's ability to generate effective fixes." The data suggests a hybrid approach — MagentLess-style localization followed by agent-based repair — that the paper does not empirically test but that the issue flow analysis directly motivates.

Interaction Efficiency: MopenHands Resolves in Fewer Turns, but with Higher Variance

Figure 5 shows the distribution of interaction turns for successfully resolved issues across MSWE-agent and MopenHands (MagentLess is excluded because it uses a fixed workflow, not multi-turn interaction). The key finding:

"MopenHands resolves issues in fewer turns than MSWE-agent when using GPT-4o for Java, whereas MSWE-agent requires fewer turns when resolving Python issues. However, MopenHands exhibits a rather higher degree of dispersion in the number of interaction turns compared to MSWE-agent, which is particularly evident on OpenAI-o3-mini-high."

This higher dispersion means MopenHands's efficiency is less predictable — some issues resolve quickly, others require many turns — while MSWE-agent's turn counts are more consistent. The paper interprets this as "MopenHands' performance is less stable across different issues, requiring a varying number of turns depending on the complexity or nature of the issue." For deployment planning, this matters: MopenHands may be more capable on average but less predictable in resource consumption, while MSWE-agent provides more reliable cost estimation.

Notable missing data points in Figure 5: the absence of box plots for certain model–language combinations (e.g., "MSWE-agent with Qwen2.5-72B-Instruct on C++") indicates cases where no issues were successfully resolved, so turn count distributions cannot be computed.

Performance Influencing Factors: Description Length, Patch Characteristics, and Repository Metrics

The paper analyzes three categories of influencing factors in Section 6.2.

Issue description length (Figure 10). The relationship between issue description length and resolved rate is inconsistent across languages:

"there is no consistent relationship between issue description length and resolved rate. For example, in Python, issues with longer descriptions tend to have lower resolved rates, whereas in Go, longer descriptions are associated with higher rates."

The paper hypothesizes that long descriptions have two opposing effects: they may indicate detailed, well-specified issues (helpful) or complex, underspecified issues that require extensive explanation (harmful). The net effect depends on which type dominates in a given language's dataset. MagentLess shows a relative advantage with longer descriptions: "compared with MSWE-agent and MopenHands, MagentLess generally performs better with longer, more detailed descriptions on average of the nine LLMs." This aligns with MagentLess's structured workflow — detailed descriptions provide clearer signals for fault localization — while agent-based methods may struggle to effectively utilize long contexts.

Fix patch length (Figure 13). A clear negative correlation emerges: resolved rates drop sharply for patches exceeding 600 tokens:

"issues with descriptions >600 tokens exhibit a resolved rate approximately 50% lower than that of issues with descriptions <200 tokens."

The paper uses "descriptions" here to mean fix patches (the section heading is "Characteristics of Fix Patches"). For Java, patches >1,000 tokens drop to 0% resolved rate (Figure 13b, MagentLess bar absent, MSWE-agent and MopenHands bars at zero). The paper interprets this as evidence that "long patches, which likely require handling a larger scope of modifications, present greater challenges, especially for methods that may not be optimized for such complex tasks."

Number of files modified by fix patches (Figure 14). Multi-file fixes consistently reduce resolved rates:

"resolved rate drops significantly as the number of modified files increases across all three methods."

Single-file fixes are where MagentLess shines: "For issues resolved by modifications in a single file, MagentLess outperforms MSWE-agent and MopenHands in five out of seven programming languages." This reinforces MagentLess's strength in targeted, localized repairs and its weakness in cross-file reasoning.

Repository quality (Figures 6–7). Repositories with higher star counts, fork counts, issue counts, and PR counts tend to show higher resolved rates, particularly for MSWE-agent and MopenHands:

"repositories with greater activity and community engagement (i.e., higher counts of stars, forks, issues, and PRs) are typically associated with a higher resolved rate."

MagentLess shows lower variance across repository quality metrics — its resolved rate is less sensitive to repository characteristics — which the paper attributes to its fixed workflow being less dependent on the richness of repository metadata.

Repository complexity (Figure 8). Three complexity metrics are analyzed: lines of code (#LoC), number of files (#Files), and language entropy (defined as $H(L) = -\sum_{i=1}^n p_i \log(p_i)$ where $p_i$ is the proportion of code in language $l_i$ ). The trend is negative for all three:

"All three methods exhibit fluctuations in performance with changes in #LoC, #Files, and language entropy, generally decreasing as the repository complexity increases."

Language entropy shows the clearest negative trend: "repositories with lower entropy typically achieve higher resolved rates." This suggests that multi-language codebases (e.g., a TypeScript project with embedded JavaScript, or a C++ project with Python scripting) are particularly challenging for current agents, likely because they require context-switching between language paradigms within a single issue resolution.

Ablation Studies and Robustness Checks

Multi-method evaluation across nine models: Not a traditional ablation, but the breadth of model coverage (spanning proprietary frontier models, open-weight models, and reasoning-specialized models across three price tiers) provides a robustness check on the core finding that non-Python languages are substantially harder. The pattern holds across all models — even Claude-3.7-Sonnet, the best performer — confirming that the Python-to-multilingual gap is not an artifact of any single model's training data distribution (Table 4).

Difficulty stratification (Table 5): The three-level difficulty breakdown tests whether aggregate performance is driven disproportionately by easy instances. Table 5 confirms that hard issues are essentially unsolved across all models, methods, and languages, while easy issues show substantially higher resolved rates (e.g., Claude-3.7-Sonnet + MopenHands achieves 48.15% on easy Java vs. 0% on hard Java). This validates the time-based difficulty categorization as capturing genuine complexity variation.

Issue type analysis (Table 6): Breaking down Claude-3.7-Sonnet's performance by issue type (Bug Fix, New Feature, Feature Optimization) tests whether the performance hierarchy is task-dependent. The consistent "Bug Fix > New Feature > Feature Optimization" ranking across languages suggests that current agents are specialized for corrective tasks and struggle with constructive or optimization-oriented tasks that require intent understanding and multi-component reasoning.

Python as an implicit ablation: The inclusion of Python results alongside the seven Multi-SWE-bench languages serves as a control. If cross-language performance drops were purely due to method adaptation issues (Section 5.1 modifications), we would expect the gap to be consistent across languages with similar adaptation difficulty. Instead, we observe large variation in the gap — Java (2× drop), TypeScript (10–20× drop), Go (5–10× drop) — suggesting language-specific challenges beyond method suboptimality.

Missing ablation: adaptation quality vs. genuine capability limitations. A critical ablation not performed is evaluating the original Python-optimized methods (without multilingual adaptations) on non-Python languages to measure the marginal contribution of the adaptations. Without this, we cannot distinguish between "the methods are poorly adapted" and "the LLMs genuinely lack multilingual software engineering capability." The paper acknowledges this limitation implicitly by noting "substantial room for improvement" in method adaptation, but does not attempt to quantify how much of the performance gap could be closed by better adaptation.

Missing ablation: model scale effects. All evaluations use the largest available version of each model family, so there is no within-family scaling analysis (e.g., comparing Qwen2.5-72B vs. Qwen2.5-7B, or Claude-3.5-Sonnet vs. Claude-3.5-Haiku). Such an analysis would reveal whether multilingual issue resolving follows predictable scaling laws or whether capability thresholds exist at particular model sizes.

Missing ablation: few-shot vs. zero-shot prompting. The paper does not specify whether the LLMs receive few-shot examples (and if so, whether they are language-specific or shared across languages). This is a potentially large confound: if the Python methods used Python-specific few-shot examples that were translated for other languages, the quality of few-shot demonstration may vary by language.

Critical Assessment

Claim 1: Existing LLMs and methods demonstrate strong performance in resolving Python issues but struggle to generalize effectively across other languages.

This claim is strongly supported by the aggregate results in Table 4 and the difficulty-stratified results in Table 5. The Python-to-multilingual gap is large (2× to 23×), consistent across all nine models, and not attributable to any single method's peculiarities since it appears under MagentLess, MSWE-agent, and MopenHands. However, the claim's strength is qualified by the Python comparison not being controlled: the Python methods (original Agentless, SWE-agent) benefit from optimization and iteration that the adapted methods (MagentLess, MSWE-agent) do not, making the gap potentially exaggerated. The paper acknowledges this (Section 6.4: Python uses original SWE-agent without truncation), but the reader should understand that the 2–23× gap represents an upper bound on the true capability difference — the real gap, with equally optimized methods, might be smaller. How much smaller is unknown and is a key open question.

Claim 2: Performance is sharply modulated by human-labeled difficulty, with hard issues showing near-zero resolved rates.

This claim is strongly supported by Table 5, with the important qualification of the Rust outlier (14.89% hard resolved rate with Claude-3.7-Sonnet + MopenHands). The "near-zero" characterization holds for most language–method combinations but requires acknowledging this exception. The paper does not explain the Rust outlier; possible explanations include the previously noted pattern of large-scale but mechanically simple Rust edits (inflating the "hard" time estimate relative to actual required reasoning) or genuine Rust-specific capability advantages in particular models.

A more fundamental concern: the difficulty labels are estimated by annotators (who may not have solved the issue), not measured from actual resolution times. The reliability of these estimates is unknown — the paper reports no inter-annotator agreement metrics for difficulty estimation. If annotators systematically under- or over-estimate difficulty for certain languages, the difficulty-stratified results would be misleading. The 80% accuracy threshold for annotation quality applies to the filtering questionnaire (Q2.1, Q3.1, Q4.1), not to difficulty estimation, which is a separate field in the annotation form.

Claim 3: The Multi-SWE-bench construction pipeline ensures diversity, executability, and human-verified correctness.

This claim is about benchmark quality rather than LLM performance, and the evidence is mixed. Diversity is well-supported: 39 repositories spanning 7 languages, from 6.7k to 698.6k lines of code, with high and low language entropy. Executability is supported by the Docker construction process (Phase 3) and the three-configuration test validation (Phase 4), though the paper does not report how many PRs failed at the environment determination stage, which would reveal the yield of the pipeline. Human-verified correctness is supported by the dual-annotation protocol with cross-review and the 80% accuracy threshold, but the paper does not report actual inter-annotator agreement metrics (Cohen's kappa, percent agreement) or the accuracy of outsourced annotations against the internal team's reference answers beyond stating the 80% threshold was met. Without these metrics, the reader cannot assess how much uncertainty remains in the instance labels.

Additionally, the filtering criterion (Q2.1=0 & Q3.1∈{2,3} & Q4.1∈{2,3}) retains only instances scoring 2–3 on clarity and test coverage. This means the benchmark is biased toward well-specified, well-tested issues — exactly the kind of issues that are easier for LLMs. Real-world software engineering includes ambiguous, poorly specified, and partially tested issues. Multi-SWE-bench's quality filtering makes it a clean evaluation surface but potentially an unrepresentative one for deployment scenarios where issue quality is lower.

Claim 4: Agent-based methods (MopenHands, MSWE-agent) generally outperform the fixed-workflow method (MagentLess).

The claim holds for most model–language combinations but with important exceptions. MopenHands wins in five of seven languages, but MSWE-agent wins on Java and TypeScript (with Claude-3.7-Sonnet), and MagentLess is competitive or better for weaker models (DeepSeek-R1, Qwen2.5-72B-Instruct). The claim also ignores an important dimension: fault localization accuracy. Figure 4 shows that MagentLess consistently achieves better localization than the agent-based methods, meaning the "outperformance" is driven by repair quality, not by the full resolution pipeline. For applications where localization is the primary bottleneck (e.g., very large codebases where agents get lost), MagentLess might be preferable despite lower overall resolved rates.

What would strengthen the paper:

Controlled Python comparison: Evaluate the adapted methods (MagentLess, MSWE-agent) on the original Python SWE-bench Verified instances to measure how much performance is lost purely due to method adaptations. This would isolate the Python-to-multilingual gap that is genuinely due to language difficulty versus the gap due to method suboptimality.
Inter-annotator agreement for difficulty labels: Report kappa or percent agreement for the time-based difficulty estimation, ideally per-language, to enable readers to assess the reliability of the difficulty-stratified analyses.
Variance estimates: Report standard deviations or confidence intervals for resolved rates, particularly for per-language and per-difficulty breakdowns where sample sizes are small (e.g., N=10 for easy JavaScript, N=30 for easy C).
Within-family model scaling: Evaluate a smaller model from the same family (e.g., Qwen2.5-7B, Claude-3.5-Haiku) to test whether multilingual capability scales predictably with model size or whether specific capability thresholds exist.
Language-native method baselines: If any issue-resolving agents exist natively for non-Python languages (e.g., tools designed for Java or JavaScript), including them as baselines would contextualize the adapted methods' performance.
Difficulty estimation ablation: Test whether annotator-estimated difficulty correlates with actual LLM success rates (which it clearly does for hard issues) and whether time-based difficulty is a better predictor of LLM performance than metric-based alternatives (patch size, file count). The paper argues this conceptually but provides no quantitative comparison.

6. Limitations and Trade-offs

Python as an Uncontrolled Reference Point

The assumption or constraint. The paper treats Python resolved rates (Tables 4–5) as a reference point for comparing against Multi-SWE-bench languages, but the Python evaluations use the original method implementations (Agentless, SWE-agent) without the modifications applied to their multilingual counterparts (MagentLess, MSWE-agent). Section 6.4 acknowledges this explicitly for MSWE-agent:

"we maintain the original SWE-agent implementation for Python, which does not incorporate the over-length truncation mechanism applied to other languages."

More broadly, the paper does not evaluate the adapted methods on Python SWE-bench instances, meaning there is no controlled measurement of how much performance is lost purely due to method adaptation versus genuine cross-language capability gaps.

The consequence. The reported Python-to-multilingual gaps — ranging from roughly 2× (Java) to 20× (TypeScript) — confound two effects: (1) the intrinsic difficulty of non-Python languages for LLMs, and (2) the degradation caused by adapting Python-optimized methods through prompt revision, observation truncation, removal of the patch selection stage (MagentLess), and language-specific bug fixes. A practitioner deciding whether to deploy an issue-resolving agent on a non-Python codebase cannot determine whether the poor performance reflects fundamental LLM limitations (suggesting deployment is premature) or suboptimal adaptation (suggesting engineering investment in language-native methods could close much of the gap). The 2–20× gap should be understood as an upper bound on the true capability difference, but the magnitude of the confound is unmeasured.

What evidence exists in the paper. The adaptation modifications are described in Section 5.1. MagentLess's removal of the candidate patch selection stage (which exists in Agentless and filters incorrect patches) is a known capability reduction. MSWE-agent's observation truncation may discard information that the original SWE-agent retains for Python. MopenHands's fix for tab character rendering in git diff output (Section 5.1) reveals that Python-first tools embed assumptions (whitespace is cosmetic) that are false for other languages (Go uses tabs semantically). These are individually documented, but their collective impact on resolved rates is never isolated or measured.

Mitigation status. The paper acknowledges the limitation partially in Section 6.1.2:

"there remains substantial room for improvement, particularly in language-specific adaptation and overall robustness."

However, this acknowledgment is qualitative and aspirational. No ablation exists that would quantify the adaptation gap — for example, evaluating MagentLess on Python SWE-bench instances to measure performance loss relative to Agentless, or running the original (non-truncated) SWE-agent on a non-Python language where truncation is not needed. The paper treats the adapted methods as the evaluation baseline and leaves method improvement to future community work, which is reasonable for a benchmark paper but leaves a critical uncertainty in the headline finding.

The Hard-Issue Ceiling: Current Methods Are Useless on Complex Tasks

The constraint. The paper's difficulty-stratified results (Table 5) demonstrate that across all models, methods, and languages, issues categorized as "hard" (estimated human resolution time ≥1 hour) show resolved rates "approaching zero." Section 6.1.1 states:

"existing LLMs and agents are mostly ineffective, with resolved rates approaching zero. This phenomenon indicates the limitations of these LLMs and agents: they are primarily capable of addressing issues that human developers can resolve in under 15 minutes and are insufficient for handling more complex tasks requiring over one hour of human effort."

This is a hard capability bound: Multi-SWE-bench reveals a ceiling on what current LLM-based agents can achieve, and that ceiling sits at roughly 15 minutes of human-equivalent problem-solving.

The consequence. For practitioners, this finding has direct deployment implications. If a target codebase's issue distribution contains a substantial fraction of complex problems (multi-file reasoning, new feature implementation, cross-component refactoring, optimization requiring deep system understanding), current LLM-based agents will fail on those problems regardless of model choice, method choice, or compute budget. The agents can handle the "routine" issues that a developer fixes before lunch; they cannot handle the "hard" issues that occupy an afternoon. This is not a gradual degradation — it is a cliff. Section 6.2.3 reinforces this: resolved rates drop ~50% when fix patches exceed 600 tokens, and drop sharply when modifications span more than one file. Hard issues compound these challenges (Table 2: hard Java issues average 246.1 lines and 5.4 files per fix patch, hard C++ issues average 763.7 lines and 11.1 files), meaning the hard-issue failure is likely driven by the multi-file, large-scope nature of complex repairs rather than any single capability gap.

What evidence exists in the paper. Table 5 provides per-difficulty per-language per-method resolved rates. Scanning the Hard columns: most entries are 0.00%. The highest is 14.89% for Claude-3.7-Sonnet + MopenHands on Rust — a notable outlier the paper does not explain, though it may relate to the observation in Section 3.2 that "certain easy instances exhibit large-scale edits (e.g., Rust), which are typically due to highly repetitive and pattern-consistent changes," suggesting some Rust instances categorized as "hard" by time estimate may be mechanically solvable pattern-consistent refactors rather than conceptually complex. Excluding this outlier, hard-issue resolved rates are 0–3% across languages. The difficulty distribution of the benchmark itself (computed from Table 2) shows that hard issues constitute a substantial fraction of the dataset: roughly 28% of Java instances, 29% of TypeScript, 41% of JavaScript, 30% of Go, 20% of Rust, 34% of C, and 33% of C++ are hard. The benchmark is not skewed toward easy instances; it includes a realistic proportion of complex tasks.

Mitigation status. The paper does not attempt to address this limitation through method design. It frames the hard-issue failure as motivation for its RL community initiative:

"This finding further underscores the need for RL techniques aimed at advancing agents towards more human-like intelligence, particularly for tackling real-world complex scenarios."

This is forward-looking but speculative — there is no evidence in the paper that RL (or any other technique) will bridge the gap from 15-minute to multi-hour problem-solving. The hard-issue ceiling may reflect fundamental architectural limitations of current LLMs (context retention across large patches, multi-file reasoning, causal understanding of complex system behavior) that incremental training improvements cannot overcome.

Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Numbers

The constraint. The paper's difficulty categorization, while analytically valuable, requires human annotation during benchmark construction — it is not available to LLMs at inference time. Unlike the paper analyzed in the reference example (which developed a PRM-based difficulty estimator for adaptive test-time compute allocation), Multi-SWE-bench provides no mechanism for an agent to estimate instance difficulty before attempting resolution. The difficulty labels exist for analysis but not for deployment.

The consequence. The paper's finding that "easy issues show substantially higher resolved rates" (Table 5) cannot be leveraged by a deployment system to route easy issues to agents and hard issues to humans. A deployed system must attempt every issue without knowing its difficulty, meaning the aggregate resolved rates in Table 4 (which average across easy, medium, and hard instances) represent the actual expected performance. The difficulty-stratified analysis identifies where the capability bound lies but does not provide a mechanism for operating below it. This contrasts with approaches that estimate difficulty from lightweight signals (e.g., verifier scores on a few initial samples) and allocate compute adaptively — a technique that would require infrastructure Multi-SWE-bench does not provide.

Furthermore, the difficulty annotations themselves depend on human estimates of resolution time, which introduces a reliability concern. The paper reports no inter-annotator agreement metrics for difficulty estimation (the 80% accuracy threshold applies to the filtering questionnaire — Q2.1, Q3.1, Q4.1 — not to the time estimate). If annotators disagree substantially on whether an issue takes 15 minutes versus 45 minutes, the easy/medium/hard boundaries blur, and the "hard-issue near-zero resolved rate" finding becomes less crisp. This is a measurement reliability concern that affects the strength of the difficulty-stratified conclusions.

What evidence exists in the paper. The difficulty distributions (Table 2, Figure 3) show substantial variation across languages: JavaScript has 241 hard instances but only 10 easy ones; Go has 141 easy, 153 medium, 134 hard. This variation could reflect genuine differences in repository complexity or annotator bias (e.g., JavaScript annotators systematically estimated higher times than Go annotators). The paper provides no within-language or cross-language calibration of time estimates, making it impossible to distinguish these explanations.

Mitigation status. Not addressed. The paper treats difficulty labels as ground truth for analysis purposes, which is standard for benchmark construction — the labels inform our understanding of model behavior, not the agent's decision-making. However, the paper does not flag this as a limitation or suggest that future work should develop automated difficulty estimation for deployed systems. This is a missed opportunity, given that the analysis itself identifies difficulty as the dominant factor in performance.

Single Benchmark Domain (Math/Logic Reasoning Analog: Repository-Level Bug Fixing Only)

The constraint. All evaluations are conducted on a single task — issue resolving on GitHub repositories — using a single benchmark (Multi-SWE-bench, with SWE-bench Verified as the Python reference). The paper does not evaluate performance on other software engineering tasks (code generation, code translation, documentation, refactoring), other evaluation paradigms (unit test generation, code review), or other repository hosting platforms (GitLab, Bitbucket). Section 7 acknowledges this scope limitation implicitly:

"Beyond issue resolving, we would like to incorporate a broader range of software engineering tasks into our benchmark and RL community, such as end-to-end project generation, runtime environment setup, bug reproduction and localization, and software testing and maintenance."

The consequence. The paper's findings — e.g., that LLMs perform well on Python but poorly on TypeScript, that bug fixes are easier than feature optimization, that hard issues are unsolved — are specific to the issue-resolving task as formulated by SWE-bench. A practitioner deploying an LLM-based agent for a different software engineering task (e.g., generating unit tests for a Java codebase, or translating Python code to Rust) cannot directly extrapolate from Multi-SWE-bench results. The benchmark evaluates diagnosis and repair of existing code; it does not evaluate construction of new code, understanding of code without an associated bug, or interaction with human developers during the development process. The issue-resolving task is a specific (and important) slice of software engineering, but it is not the whole discipline, and the paper's conclusions about language difficulty hierarchies (Python > Java > Go/Rust > C/C++ > TypeScript/JavaScript) may not generalize to other tasks.

More subtly, the benchmark's construction pipeline filters for repositories with CI/CD support, build viability, test coverage, and clear issue descriptions — criteria that exclude substantial categories of real-world software (legacy codebases without CI, research code with ad-hoc build processes, embedded systems with hardware-dependent tests, closed-source repositories). The 39 repositories in Multi-SWE-bench are high-quality, well-maintained open-source projects by construction. LLM performance on less structured repositories — where issue descriptions are terse, test coverage is sparse, and build processes are undocumented — may be substantially worse than the already-low resolved rates reported in Table 4.

What evidence exists in the paper. All results (Tables 4–8, Figures 3–14) are drawn from the 1,632-instance Multi-SWE-bench test set and the Python SWE-bench Verified reference. There is no cross-benchmark correlation analysis. The paper does not test whether model performance on Multi-SWE-bench correlates with performance on, say, HumanEval across languages, or whether the language difficulty ranking is consistent across task types.

Mitigation status. The paper acknowledges the scope limitation in Section 7 as future work, but does not attempt to validate the generality of its findings against any external task or benchmark. This is standard for a benchmark paper introducing a new evaluation surface, but it means the paper's conclusions should be understood as statements about issue-resolving capability specifically, not about multilingual software engineering capability broadly.

No Statistical Rigor: Single-Pass Evaluation Without Variance Estimates

The constraint. The paper reports resolved rates as point estimates from a single evaluation pass across the 1,632 instances, with no confidence intervals, standard deviations, or statistical significance tests. Per-language sample sizes range from 128 (Java) to 590 (JavaScript), and per-difficulty-bin sample sizes can be extremely small — 10 instances for easy JavaScript, 27 for easy Java, 28 for easy C++ (Table 2). The paper does not address how these small sample sizes affect the reliability of point estimates, nor does it perform any resampling or bootstrap analysis to quantify uncertainty.

The consequence. Many of the paper's fine-grained claims rest on comparisons between point estimates with unknown variance. For example, Section 6.1.2's claim about MagentLess outperforming MSWE-agent for DeepSeek-R1 and Qwen2.5-72B-Instruct "for languages except C and C++" is based on comparing resolved rates that may differ by only a few percentage points on small per-language samples. With 128 Java instances and resolved rates in the 5–22% range, a 3-percentage-point difference represents roughly 4 additional resolved instances — well within the range of sampling variability. Without variance estimates, the reader cannot distinguish between genuine method-level differences and noise.

The small per-difficulty-bin sample sizes are particularly concerning for the paper's central finding about hard issues. The claim that hard-issue resolved rates are "near zero" rests on bins containing as few as 27 instances (easy Java) to 241 instances (hard JavaScript). For the smaller bins, observed zero or near-zero resolved rates could be consistent with a true resolved rate of, say, 5–10% if sampling variability is high. The paper's qualitative framing ("approaching zero") is appropriate given this uncertainty, but the point estimates in Table 5 (e.g., 3.13% for OpenAI-o1 + MSWE-agent on hard Java) are reported with misleading precision.

What evidence exists in the paper. None. The paper reports no statistical methodology beyond the mean aggregation of resolved rates. The dual-annotation quality assessment (80% accuracy threshold) provides some quality control on instance labels but no statistical framework for evaluating result reliability.

Mitigation status. Not addressed. This is a significant methodological weakness relative to the paper's ambitions. The fine-grained, multi-dimensional analysis (by language, difficulty, issue type, description length, patch characteristics, repository complexity) multiplies the number of comparisons without any correction for multiple testing or any quantification of uncertainty. The paper's conclusions about, for example, the relationship between issue description length and resolved rate (Figure 10, Section 6.2.2 — "there is no consistent relationship") may reflect genuine heterogeneity or may reflect noise from small, unevenly distributed sample sizes across the token-length bins. Future work should report bootstrap confidence intervals for resolved rates, particularly for per-language and per-difficulty breakdowns, to enable readers to assess which of the paper's many claims are statistically reliable.

The RL Community Infrastructure Bet Is Unvalidated

The assumption. The paper launches the Multi-SWE-RL community with an initial dataset of 4,723 instances and positions it as foundational infrastructure for "scaling RL in real-world software environments" toward AGI (Section 4). This positioning assumes that (1) the automated pipeline (Phases 1–4 without manual verification) produces instances of sufficient quality for RL training, (2) RL with test-pass/fail reward signals will substantially improve multilingual issue-resolving capability, and (3) the community contribution model will produce sustainable dataset growth.

The consequence. If these assumptions fail, Multi-SWE-RL may not deliver on its promise — and the paper's framing of its contribution as a catalyst for RL-driven progress may prove premature. Specific risks include:

Training data quality without manual verification: The 4,723 Multi-SWE-RL instances bypass Phase 5 (manual verification), meaning they lack the quality filtering that ensures clear issue descriptions, adequate test coverage, and absence of serious issues (Q2.1=0, Q3.1∈{2,3}, Q4.1∈{2,3}). An RL agent trained on these instances may learn to exploit test suite artifacts, produce patches that pass tests without resolving the described issue, or fail silently when issue descriptions are ambiguous. The paper provides no characterization of how unverified instance quality compares to verified instance quality — for example, what fraction of Multi-SWE-RL instances would pass the manual verification filter if annotated?
RL reward signal sparsity: The very low resolved rates on non-Python languages (Table 4) mean that an RL agent operating on Multi-SWE-RL instances will receive positive reward (test pass) very rarely during early training — potentially too rarely for effective credit assignment. The paper does not discuss reward shaping, curriculum design, or other RL-specific techniques needed to overcome this sparsity.
Community contribution sustainability: The paper's incentive model (co-authorship on quarterly arXiv updates for contributed instances) is novel but unvalidated. Academic benchmark communities have historically struggled to attract sustained contributions after the initial paper's publication, and the administrative overhead of reviewing contributions, maintaining quality standards, and adjudicating authorship may exceed the paper's ongoing resources.

What evidence exists in the paper. None. The Multi-SWE-RL release is described (Section 4) but not evaluated — no RL experiments are reported, no quality analysis of the unverified instances is provided, and the community contribution model is aspirational with no track record. The paper is transparent about this: Multi-SWE-RL is positioned as an "initial contribution" and a "spark—igniting broader community collaboration." However, the paper's framing of Multi-SWE-RL as a core contribution (listed third among three main contributions in Section 1) creates an expectation of validation that the paper does not meet.

Mitigation status. The paper acknowledges the multi-SWE-RL release as preliminary:

"we release a dataset of 4,723 containerized issue-resolving instances spanning 7 programming languages. Each instance is equipped with a reproducible execution environment, enabling plug-and-play training for RL agents in realistic software contexts."

This is an honest description of the release's scope, but it does not address the quality question. The paper could have strengthened this contribution by reporting, for example, a random-sample manual verification of 100 Multi-SWE-RL instances to estimate the fraction that meet Multi-SWE-bench quality standards, or by conducting a small-scale RL experiment to demonstrate feasibility. Without such evidence, Multi-SWE-RL is best understood as a dataset release rather than a validated training infrastructure, and its impact depends on future work that the paper does not conduct.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the landscape of LLM-based software engineering by making a simple but consequential measurement: Python issue-resolving capability does not transfer to other programming languages. The measurement itself is not conceptually complex — it is a benchmark — but its effect on the field's self-understanding is substantial. Before Multi-SWE-bench, the rapid progress on SWE-bench Verified (0.40% → 65.40% in under a year) created a narrative of approaching competence: LLMs were becoming autonomous software engineers. This narrative implicitly generalized from Python to software engineering as a discipline. Multi-SWE-bench demonstrates that this generalization is false, and by a wide margin: the best available model (Claude-3.7-Sonnet) achieves 45.80% on Python (MSWE-agent) but drops to 23.44% on Java, 11.16% on TypeScript, 4.78% on JavaScript, and 5.37% on Go (Table 4). These are not minor degradations — they represent a 2× to 10× performance cliff depending on the language.

The shift this causes is epistemological: the field can no longer report progress on "automated software engineering" without specifying the programming language. A paper that improves SWE-bench by 5% has demonstrated improved Python issue-resolving capability. Whether that improvement extends to Java, Go, or TypeScript is an open empirical question that Multi-SWE-bench now enables researchers to answer. This is analogous to the shift in NLP when researchers recognized that English-only benchmarks (GLUE, SuperGLUE) did not measure general language understanding, and multilingual benchmarks (XTREME, XGLUE) revealed systematic capability gaps. Multi-SWE-bench does for code evaluation what those benchmarks did for language evaluation: it exposes that the evaluation surface was monocultural in ways that produced a misleading picture of overall capability.

The magnitude of the shift is diagnostic, not prescriptive. The paper does not propose a method, a training recipe, or a new architecture that would close the identified gap. It provides measurement infrastructure and baseline results. This makes the contribution a reframing rather than a paradigm shift: the field's goals (autonomous software engineering) remain the same, but Multi-SWE-bench reveals that those goals require multilingual capability that current methods do not possess. The reframing has immediate consequences for how researchers should design experiments and interpret results:

Research that is only evaluated on SWE-bench becomes harder to contextualize. A paper claiming to improve "code repair" that only reports SWE-bench numbers has left the most important generalization question unaddressed. Multi-SWE-bench provides the tool to address it, and the baseline results (Tables 4–5) provide the reference point against which future multilingual claims should be measured. The 23.44% ceiling on Java with the best current model and method is the number to beat.
Method design assumptions are surfaced as language-specific. The adaptations described in Section 5.1 — file skeleton removal, Tree-sitter integration, .gitignore for compiled artifacts, tab rendering fixes — reveal that the original Python methods (Agentless, SWE-agent, OpenHands) embed assumptions that break for other languages. These are not surface-level issues; they reflect architectural choices (fixed workflows, agent-computer interfaces, observation handling) designed around Python's ecosystem. The paper implies, without explicitly stating, that genuinely multilingual methods may require fundamentally different architectures — not just prompt translations and artifact filtering.
The difficulty ceiling becomes a first-order constraint on deployment expectations. The finding that hard issues (≥1 hour of estimated human effort) show near-zero resolved rates across all models, methods, and languages (Table 5) establishes a hard capability bound: current LLM-based agents can handle tasks solvable by humans in under 15 minutes, and nothing beyond that. This bound is invariant across models — even Claude-3.7-Sonnet and OpenAI-o1, the strongest performers, cannot touch hard issues — suggesting it reflects a fundamental limitation of current architectures rather than a training data or fine-tuning gap. The implication for deployment is that LLM-based agents can augment developers on routine fixes but cannot replace them on complex ones, and the boundary between "routine" and "complex" corresponds roughly to 15 minutes of human cognitive effort.

The paper also resolves a latent contradiction in how the field interprets SWE-bench progress. For researchers working primarily on Python, the rapid improvement from 0.40% to 65.40% on SWE-bench Verified represents genuine progress — methods are getting better at Python issue resolving. For skeptics who see LLMs as pattern matchers rather than reasoning engines, those gains represent overfitting to Python's specific characteristics (interpreted execution, dynamic typing, rich test infrastructure) with no guarantee of generalization. Multi-SWE-bench provides evidence for both positions: the methods are genuinely improving (they achieve non-trivial resolved rates on Java, Go, and Rust), but the improvement is dramatically Python-specific (the gap is 2× to 20× depending on language). This reconciliation is valuable because it reframes the debate from binary (generalization vs. no generalization) to continuous (generalization exists but is sharply attenuated, and the attenuation varies by language domain).

Research directions that become more attractive:

Language-native agent architectures. The paper's adaptation approach — taking Python methods and patching them for other languages — is a reasonable first step but its poor results (Table 4) suggest the ceiling is low. Designing agents from scratch for Java (with Maven/Gradle integration, static type awareness, annotation processing) or Go (with module-aware search, goroutine-aware debugging, interface satisfaction checking) may yield substantially better performance than porting Python agents.
Cross-language transfer learning. The performance variation across languages (Java ~23%, Go ~7%, TypeScript ~2%) suggests that some languages share more conceptual structure with Python than others. Training an agent on multiple languages simultaneously, or pre-training on a high-resource language and fine-tuning on a low-resource one, could improve aggregate multilingual capability.
Difficulty estimation for task routing. The paper's difficulty labels (Table 2) are analytic, not operational — an agent cannot know an issue's difficulty before attempting it. Developing lightweight difficulty estimators (e.g., from issue description features, repository structure, or quick initial exploration) would enable deployment systems to route easy issues to LLM-based agents and escalate hard issues to human developers, making the capability bound actionable rather than merely diagnostic.

Research directions that become less attractive:

Further optimization of Python-only methods without multilingual evaluation. Given that SWE-bench Verified is approaching saturation (65.40%) and Multi-SWE-bench shows the capability is language-specific, a paper reporting a 2% improvement on SWE-bench without corresponding Multi-SWE-bench results provides limited evidence of general progress.
Uniform test-time compute scaling without difficulty conditioning. The paper does not study test-time strategies, but its finding that hard issues show near-zero resolved rates regardless of model or method implies that simply scaling inference compute (more samples, more agent turns) will not help on hard instances — the models lack the fundamental capability, not the exploration budget. This is different from the adaptive test-time compute literature (e.g., the reference example paper in this prompt, which showed that difficulty-conditioned allocation helps on medium problems but fails on hard ones for similar reasons).

Follow-Up Research This Work Enables

Directly measuring the adaptation confound in the reported Python-to-multilingual gap. The paper acknowledges (Section 6.4) that Python evaluations use the original method implementations without the modifications applied to multilingual variants. A critical follow-up study would evaluate the adapted methods (MagentLess, MSWE-agent, MopenHands) on the original Python SWE-bench Verified instances and compare against the reported Python numbers (from original Agentless, SWE-agent, and OpenHands). The difference quantifies how much performance is lost purely due to method adaptation — removal of patch selection (MagentLess), observation truncation (MSWE-agent), prompt revision, .gitignore filtering — rather than genuine language difficulty. If MagentLess on Python achieves, say, 38% versus Agentless's 42%, then roughly 4 percentage points of the Java gap are attributable to method degradation, and the remaining ~18 points represent genuine language difficulty. This single ablation would transform the paper's headline finding from an upper bound ("at most 23.44% on Java") to a tighter estimate of the true capability gap, and would guide method developers toward the highest-leverage improvements (fixing adaptation vs. building language-native architectures).

Training language-native agents from scratch and comparing against adapted Python methods. The paper's evaluation uses Python-origin methods adapted through prompt revision and engineering patches (Section 5.1). A natural follow-up asks: if you designed an agent specifically for Java — with Maven build awareness, static type inference for fault localization, JUnit test framework integration, and Java-idiomatic repair templates — how much better would it perform than the adapted Python agents? This experiment would establish an upper bound on what language-specific optimization can achieve and would distinguish between two competing explanations for the poor multilingual results: (a) LLMs genuinely lack multilingual software engineering capability (implying the ceiling is low and only model-level improvements will help), or (b) the adapted methods are poor proxies for what language-native agents could do (implying substantial headroom exists through better tool integration). The experiment design would require building a Java-first agent parallel to Agentless or SWE-agent in architectural complexity, evaluating it on Multi-SWE-bench Java instances, and comparing against MagentLess and MSWE-agent results from Table 4. If the Java-first agent achieves 35–40% resolved rate (closer to Python performance), explanation (b) is supported and method adaptation is the bottleneck. If it achieves 25–28% (only marginally better than the adapted agents), explanation (a) is supported and LLM capability is the bottleneck.

Automated difficulty estimation for deployment-time task routing. The paper's difficulty categorization (easy/medium/hard based on estimated human resolution time) is used analytically but is not available to agents at inference time. A practical follow-up would train a lightweight classifier to predict difficulty from features observable before attempting a fix: issue description length and structure (presence of stack traces, error messages, reproduction steps), repository characteristics (language, number of files, test coverage density), and initial exploration signals (number of files matching keyword searches, output of initial grep or find commands). The training labels would be the existing difficulty annotations from Phase 5 (available for all 1,632 Multi-SWE-bench instances). The evaluation would measure (a) classification accuracy against human labels, and (b) whether routing issues based on predicted difficulty — sending easy/medium issues to an LLM agent and hard issues to a human — improves aggregate resolution throughput compared to sending all issues to the agent or all issues to humans. This experiment would convert the paper's diagnostic contribution (we can measure that hard issues are unsolved) into a deployment decision rule (we can predict which issues are hard and avoid wasting compute on them).

Cross-language transfer: pre-training an agent on high-resource languages and fine-tuning on low-resource ones. The paper's results show a clear performance hierarchy: Java (~23%) > Go/Rust (~7–16%) > TypeScript/JavaScript (~2–5%) (Table 4, Claude-3.7-Sonnet + MopenHands). This suggests that some languages share more conceptual structure with each other than with others. A parameter-efficient fine-tuning experiment could test this: train a LoRA adapter for a base agent on Java instances (the highest-performing non-Python language), then fine-tune that adapter on JavaScript instances (the lowest-performing language) and compare against training the JavaScript adapter from scratch. If the Java-pre-trained adapter achieves higher JavaScript resolved rates than the from-scratch adapter, it indicates transferable skill (perhaps around general codebase navigation or debugging strategies) that is partially language-agnostic. If there is no benefit, it suggests that the skills are language-specific (e.g., Java's static type system teaches patterns that don't apply to JavaScript's dynamic typing). The experiment would use the Multi-SWE-bench and Multi-SWE-RL instances, and the metric would be resolved rate on held-out JavaScript instances after equal training budgets in both conditions.

Verifier-guided search for patch candidate selection, with over-optimization monitoring. The paper notes that MagentLess removes Agentless's patch selection stage (Section 5.1) and that this likely reduces its resolved rate relative to what a full pipeline could achieve. A natural extension would be to implement a verifier-guided patch selection mechanism for the multilingual setting: train a process reward model (or outcome reward model) to score candidate patches based on their likelihood of passing the extracted test suite, then use best-of-N weighted selection (as in the reference example paper's PRM approach) to pick the most promising patch. The experiment would test whether this improves MagentLess's resolved rates on non-Python languages, particularly for easy and medium instances where the repair step generates multiple candidates. More importantly, it would test for verifier over-optimization across language domains: does a verifier trained on Java patch correctness generalize to TypeScript or C++ instances, or does the distribution shift cause the verifier to over-optimize and degrade performance at high sample budgets? This would connect Multi-SWE-bench to the test-time compute scaling literature and would establish whether verifier robustness is a cross-language challenge or a Python-specific one.

Hard-issue decomposition: breaking complex issues into sub-tasks that agents can solve individually. The finding that hard issues (≥1 hour estimated human effort) show near-zero resolved rates (Table 5) motivates a decomposition approach: can an LLM break a hard issue into a sequence of easier sub-issues (each estimated at <15 minutes of effort), and can an agent solve each sub-issue independently? This experiment would require defining a decomposition protocol (e.g., the LLM reads the issue description and outputs a list of file-level or function-level changes needed), executing the agent separately on each sub-task, and then combining the resulting patches. The metric would be whether the combined patch resolves the original hard issue more often than a single end-to-end attempt. Multi-SWE-bench's difficulty annotations and extracted test cases provide the evaluation surface. If decomposition works — even modestly, raising hard-issue resolved rate from near-zero to 5–10% — it would demonstrate that the hard-issue ceiling is partially a context-length and planning bottleneck rather than a fundamental capability gap, and would open the door to hierarchical agent architectures that were unnecessary in the Python-only setting but essential for multilingual deployment.

Practical Applications and Downstream Use Cases

Cost-efficient multilingual CI/CD issue triage and routing. The paper's cost analysis (Table 8) shows dramatic price variation across models: DeepSeek-V3 + MagentLess on Java costs $0.0059 per issue, while OpenAI-o1 + MSWE-agent on Python costs$ 3.75 per issue — a 635× difference. For a software organization managing multiple codebases across different languages, these numbers enable a concrete cost-allocation strategy. A CI/CD pipeline could: (1) use a lightweight model (DeepSeek-V3 or Qwen2.5-72B-Instruct) for initial fault localization on all incoming issues, leveraging MagentLess's fixed-workflow efficiency; (2) based on the difficulty signals observable at localization time (number of candidate files, search hit specificity), route likely-easy issues to the inexpensive model for repair and escalate likely-hard issues to a more capable model (Claude-3.7-Sonnet) or to human developers. The paper's difficulty-stratified results (Table 5) provide the priors for this routing: easy issues show resolved rates of 20–48% even with weaker models, so the inexpensive model captures much of the achievable value. Hard issues show near-zero resolved rates regardless of model, so escalating to a human is the correct decision regardless of model choice. The cost savings come from avoiding expensive models on issues they would fail anyway and using them only on the medium-difficulty tier where capability differences matter. At DeepSeek-V3's pricing (below $0.14 per million input tokens), a triage system handling thousands of issues per month would incur costs measured in single-digit dollars — well within the budget of even small engineering teams.

Multilingual agent benchmarking for model selection decisions. Organizations evaluating which LLM to integrate into their development workflow currently rely on Python benchmarks (HumanEval, SWE-bench) to inform that decision, implicitly assuming that Python performance predicts performance on their target language. Multi-SWE-bench provides the correction: an organization with a Java codebase should look at Table 4's Java columns, not the Python columns, when choosing between Claude-3.7-Sonnet (23.44% on Java via MSWE-agent) and DeepSeek-V3 (11.72% via MSWE-agent). The performance ranking across models is not preserved across languages — DeepSeek-R1 outperforms GPT-4o on Java (22.66% vs. 11.72% via MagentLess) but underperforms GPT-4o on Go (3.74% vs. 2.80% via MagentLess) and Rust (6.69% vs. 5.86%). A Java shop and a Go shop would make different model choices based on Multi-SWE-bench evidence, even though both would have chosen the same model based on Python benchmarks alone. This is the benchmark's most direct practical value: it replaces language-extrapolated model comparisons with language-specific ones, enabling more informed procurement and integration decisions. The cost columns in Table 8 further enable cost-effectiveness tradeoffs: an organization willing to pay $0.20/issue (Claude-3.7-Sonnet + MopenHands on Java, Table 8) for 21.88% resolved rate might instead pay$ 0.0059/issue (DeepSeek-V3 + MagentLess on Java) for 7.03% resolved rate and use the savings to hire a human reviewer for the unresolved issues, achieving higher aggregate throughput at lower cost.

Training data generation for multilingual code repair models. The Multi-SWE-RL dataset (4,723 instances with containerized environments) provides a ready-made infrastructure for training specialized code repair models through supervised fine-tuning or reinforcement learning. A practical pipeline would: (1) select a base code LLM (e.g., DeepSeek-Coder, StarCoder, CodeLlama); (2) for each Multi-SWE-RL instance, run the base model with one of the evaluation methods (MagentLess for structured repair, MSWE-agent for interactive repair) to generate candidate patches; (3) score patches by test execution (pass/fail against the extracted test suite); (4) fine-tune the base model on successful repair trajectories (issue description → correct patch) using standard SFT. The benefit over training on Python-only data is that the fine-tuned model should acquire language-general debugging and repair skills rather than Python-specific heuristics. The 76 repositories spanning 7 languages provide diversity that no single-language repair dataset can match. Multi-SWE-bench (the verified 1,632-instance subset) serves as the held-out evaluation set to measure whether the fine-tuned model genuinely improves or merely overfits to the training distribution. Given that current frontier models achieve only 5–23% resolved rates on non-Python languages (Table 4), there is substantial headroom for fine-tuning to improve performance — unlike Python, where the 65.40% ceiling limits the value of further supervised training.

When to Prefer This Method

The paper does not articulate a clear tradeoff between its benchmark and named alternatives (SWE-bench, SWE-Lancer) for a specific practitioner decision. It positions Multi-SWE-bench as a complementary extension rather than a replacement — SWE-bench for Python, Multi-SWE-bench for multilingual evaluation — and does not frame the choice as "use this benchmark instead of that one." Similarly, the methods evaluated (MagentLess, MSWE-agent, MopenHands) are compared against each other (Section 6.1.2) but not against a clear alternative method class; the paper is a benchmark and evaluation study, not a methods paper proposing a new technique.

The implicit preference the paper advocates is: when evaluating or deploying LLM-based issue resolving, do not assume Python performance generalizes — use Multi-SWE-bench (or SWE-Lancer for JavaScript/TypeScript) to measure performance on the target language directly. This is a methodological preference rather than a technical tradeoff, and it is uncontroversial given the paper's evidence. No conditional decision rules are constructed because the paper does not present competing approaches that would require a preference matrix.