StarCoder 2 and The Stack v2: The Next Generation

ArXiv: 2402.19173

🎯 Pitch

StarCoder2-15B matches or beats CodeLlama-34B—a model more than twice its size—while excelling at math reasoning and low-resource languages where even DeepSeekCoder-33B stumbles. This performance stems from a fully transparent training corpus 4× larger than the original, built from Software Heritage’s archive. The result upends the assumption that bigger models always win, showing that careful data curation and openness can close the gap.

1. Executive Summary

This technical report introduces StarCoder2, a family of open-weight Code LLMs at 3B, 7B, and 15B parameters, trained on The Stack v2—a new 900B+ token training corpus built from Software Heritage’s archive and supplementary high-quality sources including GitHub pull requests, Kaggle notebooks, and code documentation. The paper trains these models using a two-stage process (4K context base training followed by 16K long-context fine-tuning) and evaluates them across a comprehensive suite of code completion, editing, and reasoning benchmarks. The 3B model outperforms all similarly-sized competitors while matching the prior-generation StarCoderBase-15B, and the 15B model significantly outperforms CodeLlama-13B—matching or exceeding CodeLlama-34B on several benchmarks and surpassing DeepSeekCoder-33B on math reasoning (GSM8K) and low-resource languages (D, Julia, Lua, Perl), establishing that model size alone does not determine performance when training data is carefully curated and formatted with repository-level context and instruction-style pretraining objectives.

2. Context and Motivation

The Core Problem: Openness in Code LLM Development Is Fractured

The fundamental gap this paper addresses is not a single technical capability but the fractured landscape of openness in Code LLM development. As of early 2024, the field had bifurcated into two unsatisfactory camps. On one side, proprietary models like OpenAI's GPT-4 (OpenAI et al., 2023) and Google's Gemini (Gemini Team et al., 2023) offered strong performance through paid APIs but disclosed nothing about their training data, architectures, or development processes. On the other side, "open-weight" models like Code Llama (Rozière et al., 2023), Mistral (Jiang et al., 2023), and DeepSeekCoder (Guo et al., 2024) released model weights—enabling local deployment, fine-tuning, and representation inspection—but withheld their training data. The paper argues this creates three concrete harms (Section 1):

• Content creators cannot audit whether their data was used for training. Without knowledge of training data, developers who contribute to open-source repositories have no way to verify if their code (potentially under specific licenses or containing sensitive information) is being used to train commercial models.
• Social scientists cannot scrutinize datasets for bias and toxicity. Training data transparency is a prerequisite for systematic fairness audits. When datasets are opaque, researchers cannot identify representational biases (e.g., over-representation of English-language or Western programming conventions) or measure the prevalence of toxic or insecure code patterns that the model may reproduce.
• LLM developers cannot assess benchmark contamination or replicate results. When training data is secret, there is no way to verify whether a model's strong benchmark performance reflects genuine capability or inadvertent memorization of test-set solutions included in the training corpus. This undermines the scientific validity of benchmark comparisons and makes it impossible for other teams to reproduce or build upon training recipes.

The paper explicitly situates StarCoder2 within the lineage of fully open development projects—citing Allen AI's OLMo (Groeneveld et al., 2024), Eleuther AI's Pythia (Biderman et al., 2023), and BigScience's BLOOM (BigScience Workshop, 2022) as exemplars that "have adopted a fully open development approach by releasing training data, training frameworks, and evaluation suites" (Section 1). These projects demonstrated that open science is feasible for general-domain LLMs, but no project had yet applied this philosophy at scale to code-specific models while remaining competitive with closed-data alternatives.

Why Full Openness Matters Beyond Ideology

The paper's emphasis on transparency is not merely principled—it addresses concrete, practical challenges that the prior generation of Code LLMs (including BigCode's own StarCoder1) encountered.

The license attribution problem. The Stack v1 (Kocetkov et al., 2023), used to train the original StarCoder and StarCoderBase models (Li et al., 2023), restricted itself to permissively licensed code. This was a conservative, legally cautious choice, but it came at a cost: it excluded the vast majority of publicly available source code, which either carries no license or uses non-permissive (e.g., copyleft) licenses. The paper notes that in the Software Heritage archive used for The Stack v2, 96.93% of repositories lacked repository-level license information (Section 2.1). The prior approach of simply excluding unlicensed code meant discarding an enormous volume of valuable training data. The Stack v2 introduces a more nuanced license detection pipeline (using ScanCode at the file level, with a decision tree in Figure 1) that includes unlicensed files while still excluding commercial and copyleft-licensed code. This enables a 4× increase in training data volume—from ~875 GB (deduplicated) in The Stack v1 to ~6,457 GB in The Stack v2 (Table 1)—without compromising the project's responsible development principles.

The opt-out and governance gap. StarCoder1 introduced the "Am I in The Stack" tool, allowing developers to check whether their repositories appeared in the training set and to submit opt-out requests. This was a novel governance mechanism, but its practical effectiveness depends on the scale and representativeness of the underlying dataset. With The Stack v2 being more than seven times larger than v1 in terms of raw file volume (Table 1 reports 784.3 million files in v2-dedup versus 181 million in v1-dedup), the paper had to validate that the governance infrastructure scaled appropriately. The opt-out process described in Section 3.5—where developers were given a public announcement and a deadline (November 20, 2023) to submit requests—resulted in the removal of 1,561 repositories from 91 users and organizations, totalling 22,066 files. This is a tiny fraction of the corpus (0.003% of the 654M total files), suggesting either that the opt-out tool was not widely adopted or that the vast majority of developers did not object to inclusion—a finding that itself raises important questions about the effectiveness of opt-out mechanisms as a governance strategy.

The benchmark decontamination problem. When training data is opaque, benchmark scores are ambiguous. A model might genuinely learn transferable code reasoning skills, or it might simply have memorized solutions that appeared in its training corpus. The paper addresses this directly in Section 3.3, describing a decontamination process that removes files containing docstrings or solutions from HumanEval, MBPP, APPS, GSM8K, and DS-1000 from the training set. Critically, they note that for StarCoder2 they "further enhance the recall of the decontamination process by removing whitespace during string matching"—acknowledging that prior decontamination methods were insufficiently robust. Without public training data, no external evaluator could verify these decontamination claims, making benchmark results inherently less trustworthy.

Where Prior Code LLMs Fall Short

The paper positions StarCoder2 against a rapidly evolving competitive landscape and identifies specific weaknesses in prior approaches:

Code Llama (Rozière et al., 2023): At the time of StarCoder2's release, Code Llama models (7B, 13B, and 34B parameters) were the dominant open-weight Code LLMs. However, Code Llama's training data is not disclosed. The paper emphasizes this opacity as a direct limitation—researchers cannot study its training data composition, cannot verify decontamination claims, and cannot reuse its data pipeline. Furthermore, the evaluation results in Section 7 show that StarCoder2-15B consistently outperforms CodeLlama-13B across almost all benchmarks (Tables 9, 10, 11, 14, 15, 17, 18) and is competitive with CodeLlama-34B despite being less than half its size—suggesting that careful data curation and formatting may compensate for parameter count.

DeepSeekCoder (Guo et al., 2024): DeepSeekCoder models (1.3B, 6.7B, and 33B parameters) emerged as the strongest open-weight Code LLMs on many benchmarks, particularly code completion in high-resource languages. The paper acknowledges DeepSeekCoder's strengths candidly (e.g., Section 7.1.2 notes that "DeepSeekCoder-33B is the best-performing model at code completion benchmarks for high-resource languages"). However, DeepSeekCoder also does not release its training data. Moreover, the paper identifies specific weaknesses: on math reasoning (GSM8K, Table 14), StarCoder2-15B scores 65.1% versus DeepSeekCoder-33B's 58.7%; on code reasoning (CRUXEval-I, Table 15), StarCoder2-15B matches or exceeds DeepSeekCoder-33B; and on low-resource languages like D, Julia, Lua, and Perl (MultiPL-E, Table 10), StarCoder2-15B outperforms DeepSeekCoder-33B by substantial margins (D: 23.6 vs. 17.4; Julia: 33.2 vs. 32.8; Lua: 43.8 vs. 36.5; Perl: 37.2 vs. 31.0). These domain-specific advantages suggest that DeepSeekCoder's training data may be skewed toward high-resource languages and general-purpose code completion, while The Stack v2's inclusion of 619 programming languages, math datasets, and intermediate representations provides StarCoder2 with complementary strengths.

StarCoderBase (Li et al., 2023): The first-generation StarCoder models represented the prior state of fully open Code LLM development. StarCoder2 directly improves upon them along multiple axes:

• Training data volume and diversity: StarCoderBase was trained on The Stack v1, containing ~875 GB of permissively licensed code in 384 languages. StarCoder2's training set (The Stack v2) is 4× larger in terms of unique tokens (900B+ vs. ~220B for StarCoderBase) and includes unlicensed code, GitHub pull requests, issues, notebooks, documentation, and math datasets—none of which were in StarCoderBase's training data.
• Context length: StarCoderBase had a maximum context of 8,192 tokens. StarCoder2 uses 4,096 tokens for base training and then fine-tunes for 16,384 tokens using FlashAttention-2, enabling repository-level code understanding.
• Architecture improvements: StarCoderBase used learned positional embeddings and Multi-Query Attention (MQA). StarCoder2 adopts Rotary Positional Encodings (RoPE) and Grouped Query Attention (GQA), following the architectural trends established by Code Llama and DeepSeekCoder (Section 6.1).
• Repository-level training: StarCoderBase was trained with file-level context (random files joined together). StarCoder2 introduces repository-level context where files from the same repository are grouped together, preserving the directory structure and inter-file relationships that matter for real-world code understanding (Section 5.1).
• Training tokens far beyond Chinchilla-optimal: The paper explicitly states they "push the number of training tokens far beyond the compute-optimal number suggested by Chinchilla (Harm's law; de Vries, 2023) and train relatively small models within the range of 3.3 to 4.3 trillion tokens" (Section 1). StarCoder2-3B is trained on ~3.1T tokens (~5 epochs), StarCoder2-7B on ~3.5T tokens (~5.3 epochs), and StarCoder2-15B on ~4.1T tokens (~4.5 epochs)—massively over-trained relative to Chinchilla scaling laws but aligned with the data-constrained regime described by Muennighoff et al. (2023), where repeating data for multiple epochs can yield continued improvements when data is limited relative to model capacity.

The Missing Piece: Repository-Level Context and Instruction-Style Pretraining

Beyond transparency, the paper identifies a specific technical gap in how Code LLMs are trained: most models are trained on isolated code files without repository context. Real-world software development involves navigating multi-file codebases, understanding dependencies, and fixing bugs across module boundaries. Prior models like StarCoderBase, Code Llama, and DeepSeekCoder did not systematically train on repository-structured data—they treated code files as independent documents.

StarCoder2 addresses this through two novel data formatting choices:

1. Repository-context source code (Section 5.1): Instead of random file concatenation, files from the same repository are grouped together with their file paths. The format <repo_name>reponame<file_sep>filepath1\ncode1<file_sep>filepath2\ncode2 ... teaches the model to associate code with its location in a project structure. The paper also applies Fill-in-the-Middle (FIM) transformations at the repository level, with a 50% probability of applying FIM to each repository and, within that, a 50% probability per file chunk.

2. Pull requests as instruction-style training (Sections 2.3, 5.2, and 7.2.1): The inclusion of GitHub pull requests with their full conversation history—including code review comments, diff hunks, approval/rejection decisions, and multi-round revisions—effectively serves as a form of instruction tuning during pretraining. The format carefully structures PRs as temporal sequences: base files → initial code changes → review comments → additional diffs → merge/rejection decisions. This teaches the model to understand code modification requests, respond to reviewer feedback, and track multi-step edits—capabilities that traditional code completion models lack. The paper leverages this at evaluation time through the "Issue" prompt format (Section 7.2.1), where bug-fixing tasks are formatted in the style of a GitHub issue discussion, yielding dramatic improvements over standard instruction prompts for a base model (e.g., on HumanEvalFix, StarCoder2-15B with the Issue prompt achieves 38.7% average pass@1 versus 22.0% with a standard instruct prompt—Table 12).

Reconciling the "More Data vs. Better Data" Tension

The paper implicitly addresses a debate in the LLM community about whether data quality or data quantity matters more. The Stack v2's approach is a hybrid: it massively scales data volume (from ~875 GB to ~6,457 GB of raw deduplicated source code) while simultaneously curating high-quality supplementary sources (documentation, pull requests, math datasets, intermediate representations). The paper's ablation in model sizes (Section 4) reveals a nuanced data strategy:

• Smaller models get less diverse data: The 3B model is trained on the-stack-v2-train-smol (525B tokens from 17 programming languages plus documentation/config languages) and excludes natural language datasets (OpenWebMath, Wikipedia, ArXiv). The rationale is that "smaller models, having limited capacity, should be exposed to a less diverse dataset"—a principle supported by multilingual NLP research showing languages compete for model capacity (Arivazhagan et al., 2019; Conneau et al., 2020; Scao et al., 2022b).
• Larger models get full diversity: The 15B model uses the-stack-v2-train-full (775B tokens from all 619 programming languages) plus all supplementary sources including natural language data, for a total of 913B+ unique tokens.

This data scaling strategy reflects an underlying hypothesis: data quality and data quantity are not independent axes—they interact with model capacity. A small model trained on too many low-resource languages may suffer from capacity dilution, while a large model trained only on high-resource languages would miss the opportunity to learn transferable reasoning skills from diverse programming paradigms. The difficulty-dependent performance patterns (e.g., StarCoder2-15B outperforming larger models on low-resource languages but not high-resource ones) partially validate this hypothesis, though the paper does not isolate the effect of data composition from other confounding factors (model architecture, training duration, etc.).

How This Paper Positions Itself

The paper explicitly positions StarCoder2 as the successor in an ongoing open-science research program, not as a one-off model release. Section 1 traces the lineage: SantaCoder (1.1B parameters, Java/JavaScript/Python only) → StarCoderBase (15B parameters, 384 languages) → StarCoder2 (3B/7B/15B parameters, 619 languages, with pull requests, notebooks, documentation, and math data). The BigCode project is framed as a "community of practice" with "over 1,200 multi-disciplinary members from more than 60 countries working towards the responsible development of large language models for code" (Section 9.1), and the paper emphasizes that 417 members were active during the StarCoder2 development period.

Critically, the paper does not claim to be the best at everything. It is notably candid about limitations and failures:

• StarCoder2-7B is acknowledged as an underperformer: "It is not clear to this report's authors why StarCoder2-7B does not perform as well as StarCoder2-3B and StarCoder2-15B for their size" (Section 1). This is an unusually honest admission in a model release paper.
• DeepSeekCoder outperforms StarCoder2 on several key benchmarks (HumanEval, MultiPL-E for high-resource languages, DS-1000 on TensorFlow), and the paper states this explicitly.
• The FIM implementation was buggy for most of StarCoder2-15B's training (Table 16 note: "Due to an implementation bug, the FIM-rate was smaller than intended for most of the training"), resulting in degraded fill-in-the-middle performance.
• StarCoder2-15B was early-stopped after 1M iterations instead of the scheduled 1.1M (Section 6.3).
• A RoPE θ configuration bug caused StarCoder2-15B to use a different base period (θ = 1e4) than the 3B and 7B models (θ = 1e5) (Table 7).

This transparency is itself a form of positioning: the paper argues by example that open development includes acknowledging mistakes, not just broadcasting successes. The governance infrastructure—the opt-out tool, the membership checking Bloom filter (Section 8), the Elasticsearch search index, the dataset card, the OpenRAIL license—constitutes a comprehensive framework for responsible open-source AI development that the paper positions as a model for other projects to follow, citing the BigCode Governance Card (BigCode collaboration et al., 2023) as a documented mechanism for "intentional governance of an open research project that future endeavors can leverage to shape their own approach" (Section 9.2).

3. Technical Approach

3.1 Reader Orientation

This paper is a systems-and-data engineering project report describing the construction of a new training dataset (The Stack v2) and a family of transformer language models (StarCoder2) trained on it. The core problem it solves is: how do you build a state-of-the-art code generation model while maintaining full transparency about the training data and responsible development practices? The "shape" of the solution is a two-part pipeline: first, assemble and clean a 900B+ token dataset from diverse code-related sources with auditable provenance, then train models at three sizes using architectural improvements and data formatting choices that teach repository-level code understanding.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major stages:

1. Data Sourcing (Section 2) — Collects raw data from Software Heritage's source code archive, GitHub issues/PRs, Jupyter/Kaggle notebooks, documentation websites, intermediate representations, and curated math/coding/NLP datasets. Each source has its own extraction and cleaning pipeline.

2. Preprocessing Pipeline (Section 3) — Applies deduplication (MinHash + LSH, Jaccard 0.7), PII redaction (StarPII model for code, custom regex for emails/usernames), benchmark decontamination (whitespace-insensitive string matching), malware scanning (ClamAV + SaneSecurity signatures), and opt-out removal (1,561 repositories from 91 users) to the raw data.

3. Data Composition and Formatting (Sections 4–5) — Selects which data subsets go to which model size (smaller models get fewer languages), formats each source with special sentinel tokens (e.g., <issue_start>, <repo_name>, <pr_diff_hunk>), applies Fill-in-the-Middle transformations, and organizes source code into repository-level context.

4. Model Architecture and Training (Section 6) — Defines transformer architectures (RoPE, GQA, details in Table 6) and trains at 4K context length for 3.1–4.1T tokens (~4–5 epochs), then fine-tunes for 200B additional tokens at 16K context length with FlashAttention-2.

5. Evaluation Suite (Section 7) — Tests models on code completion (HumanEval/MBPP with EvalPlus, MultiPL-E, DS-1000, RepoBench, CrossCodeEval), code editing (HumanEvalFix, CanItEdit), math reasoning (GSM8K with PAL), code reasoning (CRUXEval), fill-in-the-middle, security (Asleep at the Keyboard), and bias/toxicity benchmarks.

Information flows sequentially: data sources → preprocessing → formatting → training → evaluation, with size-dependent routing at the composition stage (Section 4) determining which data each model sees.

3.3 Roadmap for the Deep Dive

• First, I'll explain the data sourcing pipeline—how each data source is collected and what makes it valuable, since data curation is the paper's primary contribution.
• Second, I'll cover the preprocessing steps (deduplication, PII redaction, decontamination, malware removal, opt-outs) that transform raw data into training-ready text, as these define the "responsible development" claims.
• Third, I'll detail the data composition strategy—why different model sizes get different data mixtures and how the subsampling of programming languages works.
• Fourth, I'll walk through the data formatting scheme with its sentinel tokens and FIM transformations, since this is the mechanism by which repository structure and instruction-following capabilities are injected during pretraining.
• Fifth, I'll describe the model architecture and two-stage training process with all hyperparameters.
• Sixth, I'll explain the evaluation methodology, since the benchmark choices and prompting formats are critical to interpreting the reported results.

3.4 Detailed Sentence-Based Technical Breakdown

This is a data engineering and model training project report whose core idea is that full training data transparency, combined with careful multi-source data curation and repository-aware formatting, produces Code LLMs that compete favorably with larger but opaque models.

Data Sourcing: What Goes Into The Stack v2 and Why

The Stack v2 aggregates nine distinct categories of data, each contributing different types of knowledge to the final model. I'll explain each source's collection method, its role in the training mixture, and the design choices that matter.

Source Code from Software Heritage (Section 2.1): This is the primary dataset, representing the vast majority of training tokens. The authors use the 2023-09-06 version of the Software Heritage (SH) graph dataset—a fully deduplicated Merkle DAG representation of all source code ever archived by SH. From this, they extract the most recently crawled versions of all GitHub repositories, keeping only the main branch (identified via GHArchive metadata or if the branch name is main or master). They take only the latest commit from each main branch and deduplicate repositories based on unique content hashes (the directory_id column).

The file contents are downloaded from the SH content S3 bucket, restricted to files whose compressed size is less than 10MB. The directory structure is reconstructed recursively by joining the directory_entry table to itself (capped at depth 64) and concatenating directory and file names into full paths.

A critical design choice is the license detection pipeline, depicted in Figure 1. The paper inherits file-level license information from GHArchive when available (only 3.07% of repositories have it). For the remaining 96.93%, they use ScanCode Toolkit to detect licenses at the file level through a four-step process:

1. Find all files whose names match a license-related regex (covering patterns like LICENSE, COPYING, README, GUIDELINES, and versioned variants).
2. Apply ScanCode's license detection to those files and gather SPDX IDs.
3. Propagate detected licenses to all files sharing the same base path as the license file.
4. Classify each file as permissive, non-permissive, or unlicensed using the decision tree in Figure 1: if the GitHub-level license exists and is permissive, the file is permissive; if GitHub-level is non-permissive, it is excluded; if no GitHub license exists, ScanCode results are consulted—files with all detected licenses being permissive are included, files with any non-permissive license are excluded, files with no detected license are included as "unlicensed."

The list of permissive licenses (Appendix A.4) is compiled from Blue Oak Council-approved licenses and licenses categorized as "Permissive" or "Public Domain" by ScanCode. The paper explicitly excludes commercial and copyleft-licensed code: commercial licenses because "their creators do not intend their code to be used for commercial purposes," and copyleft licenses "due to uncertainty regarding the community's stance on using such data for LLM training and its relatively low volume." This is the key departure from The Stack v1, which excluded unlicensed files entirely—by including unlicensed files, the usable data volume increases dramatically.

Why this matters: The decision tree in Figure 1 is a concrete governance mechanism: it encodes a policy choice (include permissive and unlicensed; exclude commercial and copyleft) as an executable algorithm. This is auditable—anyone can inspect the code, verify the license list, and check whether their repository was correctly classified. The inclusion of unlicensed code is scientifically pragmatic (most code on GitHub has no license) but legally debated; the paper's transparency about this choice enables informed critique.

Language detection: The paper uses go-enry (based on GitHub's Linguist library) to detect programming languages for each file by content analysis rather than by extension alone. This detects 658 unique languages in the deduplicated dataset, some of which are later removed during visual data inspection.

Visual data inspection: The paper involves 15 annotators from the BigCode community who inspect extensions with over 1,000 files and representing over 0.5% of files in their respective languages. They examined approximately 1,000 extensions and excluded 130 (listed in Appendix A.1), which eliminated 39 programming languages entirely (listed in Appendix A.2). This step inherits annotations from StarCoderBase (which already eliminated 36 out of 300 extensions across 86 languages) and extends them to newly included languages.

Basic filters: Several automated filters remove low-quality files:

• Long line filter: Remove files with more than 100K lines (likely data or generated code). For most languages, also remove files with average line length > 100 characters or max line length > 1,000 characters. Exceptions are made for HTML, JSON, Markdown, Roff, Roff Manpage, SMT, TeX, Text, and XML, where only files with the longest line exceeding 100K characters are removed.
• Autogenerated filter: Use go-enry's is_generated function plus a regex check for "auto-generated", "autogenerated", "automatically generated", "generated automatically", or "this file is generated" in the first 5 lines.
• Alpha filter: Remove files with fewer than 25% alphabetic characters, except Motorola 68K Assembly and WebAssembly (where the threshold applies to alphanumeric characters instead, due to syntax differences).
• Encoded data filter: Detect and remove files with large blocks of Base64 ([a-zA-Z0-9+/\n=]{64,}), hexadecimal sequences ((?:\b(?:0x|\\x)?[0-9a-fA-F]{2}(?:,|\b\s*)){8,}), or Unicode escape sequences ((?:\\u[0-9a-fA-F]{4}){8,}). A file is removed if any matching substring exceeds 1,024 characters or if matched characters constitute more than 50% of the file.

Language-specific filters:

• For Text, JSON, YAML, Web Ontology Language, and Graphviz (DOT): remove files with more than 512 lines to minimize repeated tokens in data files.
• For HTML: keep only files where visible text is at least 100 characters and constitutes at least 20% of the code.
• For Text: keep only files with "requirement" in the lowercased filename, or whose filename (without extension) is one of {readme, notes, todo, description, cmakelists}.

Scale: Table 1 provides a direct comparison between The Stack v1-dedup, v2-dedup, and v2-swh-full for the 32 most popular programming languages. The Stack v2-dedup contains 6,457 GB across 784.3M files—a 7.4× increase in volume and 4.3× increase in file count compared to The Stack v1-dedup (875.85 GB, 181M files). The v2-swh-full (used for StarCoder2-15B training) is smaller at 1,922.82 GB across 528.44M files because it undergoes further subsampling (described in Section 4). Notable languages with huge increases include HTML (2,419.87 GB in v2-dedup vs. 146.76 GB in v1), Java (548.00 GB vs. 89.30 GB), JavaScript (1,115.42 GB vs. 141.65 GB), and C++ (353.89 GB vs. 50.89 GB).

Pull Requests (Section 2.3): The paper collects pull request events from GHArchive and the corresponding source code from Software Heritage. For each PR, they aggregate PullRequestEvent, PullRequestReviewEvent, PullRequestReviewCommentEvent, IssueCommentEvent, and IssuesEvent events. They extract all base and head commit IDs and retrieve the corresponding code files from Software Heritage. Since they lack direct access to commit diffs, they generate diffs by identifying changes between files at the same path across commits: files present in base but absent in head are deletions, and vice versa.

This process yields approximately 300M PRs with 15 TB of base code, of which 215M are closed PRs from approximately 24M repositories.

PR filters (applied sequentially):

1. Remove PRs opened by bots.
2. Remove PRs consisting only of comments by bots.
3. Remove PRs with non-permissive license.
4. Remove opted-out PRs.
5. Remove PRs where the base branch changes during the PR.
6. Remove PRs not approved or merged.
7. Remove PRs lacking initial diffs.

File-level filters for PR base code:

1. Remove deletion or addition files.
2. Remove files exceeding 1 million characters.
3. Remove files with alphanumeric fraction < 0.25.
4. Remove files with hexadecimal character fraction > 0.25.
5. Remove files with > 100,000 lines.
6. Remove files with average line length > 100.
7. Remove files with max line length > 1,000.
8. Remove Markdown files containing non-English text.

Title/description filtering: Exclude PRs whose title is less than 10 characters or contains "dependencies", "dependency", "depend", or "release". Exclude PRs whose description is less than 20 characters or contains "Qwiet."

Truncation: Titles truncated to 500 characters, descriptions to 80 lines (first 60 + last 20), then further truncated if still exceeding 1,000 characters.

Comment processing: Bot comments are removed; auto-generated text from email replies is stripped; usernames are anonymized; comments with fewer than 20 characters are removed unless they are PR review comments. For code review comments, full diff hunks exceeding 10,000 characters are replaced by just the filename and comment.

Subsampling for diversity: PRs are subsampled per repository to prevent large repositories from dominating. For repositories with 1 PR (after filtering), retention probability is 0.8; this linearly decreases to 0.1 for repositories with 1,000 PRs; for repositories with > 1,000 PRs, exactly 100 PRs are retained. Additionally, YAML and JSON files are subsampled at 10% retention when their file size exceeds 50% of total base files or the file path contains "pack", "lock", "yarn", "output", "swagger", "openapi", or "output."

Final volume: After all processing, 3.7M PRs remain at up to 1M characters (194 GB), or 3.3M PRs at up to 100K characters (67.3 GB). The paper opts for the 100K-character limit, which translates to approximately 25K tokens.

Why PRs matter: PRs represent a natural form of instruction-following data embedded in real software engineering workflows: a developer proposes changes, reviewers provide feedback, and code is revised iteratively. By training on this structure, the model learns to map between natural language instructions and code modifications without explicit instruction tuning. The paper explicitly validates this hypothesis in Section 7.2.1, where prompting StarCoder2-15B in an "Issue" format (mimicking GitHub issue discussions) dramatically improves its bug-fixing performance compared to a standard "Instruct" prompt format (38.7% vs. 22.0% average on HumanEvalFix, Table 12).

GitHub Issues (Section 2.2): Issues are collected from GHArchive, excluding pull requests (processed separately). Each issue consists of a series of events with actions (open, comment, close), an author username, a message, and a creation date. The processing pipeline:

1. Remove auto-generated text from email replies.
2. Remove issues with messages shorter than 200 characters.
3. Truncate long comments to a maximum of 100 lines (retaining first portion and last 20 lines), removing 17% of the volume.
4. Exclude comments from bots based on username keyword matching, eliminating 3% of issues (much less than StarCoderBase's 17%, because PRs—a primary source of bot content—are processed separately).
5. Quality filtering via user engagement: keep conversations with 2+ users. For single-user conversations, keep only if the total comment text is less than 7,000 characters (96th percentile) and there are 10 or fewer events (to exclude poor-quality or bot-originated threads). This removes 38% of remaining issues.

Usernames are anonymized by replacing them with a participant counter within the conversation (e.g., username_1 for the second participant).

Jupyter Notebooks (Section 2.4.1): Two formats are produced from the same notebooks:

Jupyter scripts: Notebooks are converted to plain code scripts using Jupytext. The programming language is identified from notebook metadata or, if unavailable, via the Guesslang library with a probability threshold ≥ 0.5. Starting from 11M notebooks, 3M are excluded due to parsing errors, and near-deduplication reduces the count to 4M notebooks.

Jupyter structured: Only notebooks explicitly marked as "Python" in metadata are kept. Consecutive Markdown blocks and code blocks are merged, producing temporal code-text pairs where each pair consists of the Markdown text immediately preceding a code block, the Python code itself, and the formatted output (if non-empty; otherwise an <empty_output> token). For consecutive code blocks with multiple outputs, only the last output is retained. After near-deduplication, 4.6M structured notebooks remain.

Kaggle Notebooks (Section 2.4.2): 3.6M notebooks are sourced from Kaggle under Apache 2.0 license. These do not include output cells—only Markdown and code cells. Cleaning steps:

1. Remove notebooks with fewer than 100 characters or syntax errors.
2. Remove templated text at the beginning (two specific templates listed in Appendix A.7), eliminating 18% of notebooks.
3. Convert to structured and script formats following the Jupyter pipeline.
4. Near-deduplication eliminates 78% of remaining notebooks, leaving 580K notebooks.

For 42% of notebooks associated with a Kaggle dataset, the dataset title and description are prepended as context. For 8.6% of notebooks that load CSV datasets, the paper downloads the actual datasets via the Kaggle API and prepends df.info() output (column names, dtypes, non-null counts, memory usage) plus four sample rows.

Documentation (Section 2.5): Documentation is collected from three sources:

Package manager documentation: The paper crawls npm, PyPI, Go Packages, Packagist, Rubygems, Cargo, CocoaPods, Bower, CPAN, Clojars, Conda, Hex, and Julia. Library names are retrieved from libraries.io, then used to search individual package managers for homepages. Documentation is extracted from README/documentation files or crawled from homepages using the website documentation pipeline. For PyPI and Conda libraries, Read the Docs is prioritized as the primary source.

PDF documentation: For R language, all PDF files from CRAN are extracted using the pdftotext library (preserving code snippet formatting). For LaTeX, documentation/tutorial/usage guide PDFs are extracted from CTAN, image-heavy PDFs are filtered out, and the rest are converted to Markdown using the Nougat neural OCR tool.

Website documentation: A curated list of 27 websites (Table 2) including MDN Web Docs, TensorFlow Docs, Linux Docs, Flutter API Reference, TypeScript handbook, and language-specific documentation (Swift, Qt, GCC, LLVM, etc.) is crawled. The pipeline: (1) extract content from HTML pages using the trafilatura library, converting to XML while removing navigation/index bars; (2) convert XML to Markdown; (3) near-deduplicate pages within each website using MinHash LSH with threshold 0.7, removing pages that are essentially just website layouts; (4) concatenate remaining pages in crawl order (breadth-first), producing a cohesive narrative. Additionally, code-relevant data is extracted from existing web crawls (RefinedWeb, OSCAR, esCorpius) by identifying programming language constructs via regex and "docs." substrings in URLs.

Free textbooks: 3,541 e-books are scraped from the Free Programming Books project by extracting all PDF links, downloading them, and extracting text via pdf2text.

Language identification for documentation: The main programming language for each document is identified either through predefined rules (when the source unambiguously corresponds to a language) or via the Guesslang library (when it does not). The resulting language distribution is shown in Figure 2, with CSS, Haskell, HTML, Perl, PHP, Julia, JSON, SQL, Markdown, TeX, Ruby, Python, Rust, JavaScript, Go, and R being the top languages.

Intermediate Representations (Section 2.6): The paper pairs source code with LLVM intermediate representation to help the model learn correspondences between high-level languages and their compiled form. The motivation is that "a shared intermediate representation might help to anchor low-resource constructs to similar ones in high-resource languages" (Section 2.6, citing Zhuo et al., 2023b). LLVM is chosen because it is the target of many compiler frontends across multiple programming languages.

The data collection sidesteps the challenge of compiling arbitrary source code (which often fails due to missing dependencies) by using self-contained compilation units from four sources:

• Rosetta Code (programming tasks with solutions in many languages)
• Codeforces solutions
• CodeNet
• Description2Code

Approximately 4M sources are compiled across C++, C, Objective-C, Python, Rust, Go, Haskell, D, Fortran, Swift, and Nim. Each source is compiled in both size-optimized (-Oz equivalent) and performance-optimized (-O3 equivalent) mode. The size-optimized IR is used 80% of the time (due to context length considerations), and performance-optimized IR is used 20% of the time (to maximize transfer from pre-training, where the model sees LLVM code in the wild, which is more likely to be in performance-optimized form). Compilation uses clang (C++, C, Objective-C), codon (Python), rustc (Rust), gollvm (Go), ghc (Haskell), ldc (D), flang (Fortran), and nlvm (Nim). Headers and platform/vendor/memory-layout-specific information are cleaned from the IR before pairing with source code.

Small High-Quality Datasets (Section 2.7): The "LHQ" (Leandro's High-Quality) collection includes:

• APPS (train): 5,000 Python programming problems, one solution per problem.
• Code Contest: 13K+ examples with solutions in Python 2/3, C++, and Java, one solution per problem and language.
• GSM8K (train): 7K+ math reasoning examples.
• GSM8K (SciRel): 110K examples with alternative reasoning paths for GSM8K questions.
• Deepmind Mathematics: 110M+ synthetic math questions across algebra, arithmetic, calculus, comparison, measurement, numbers, polynomials, and probability at varying difficulty levels.
• Rosetta Code: Over 1,100 programming tasks with solutions in as many languages as possible.
• MultiPL-T: Over 200K examples in Lua, Racket, and OCaml, constructed by automatically translating Python functions and validating correctness with unit tests.
• Proofsteps: From AlgebraicStack (used to train Lemma models), plus proofsteps-lean (3K+ examples from mathlib 4) and proofsteps-isabelle (250K+ examples built on the PISA dataset).

Other Natural Language Datasets (Section 2.8):

• StackOverflow: 11M questions from the 2023-09-14 dump. Quality filtering uses Llama-2-70b-chat-hf: 20K examples are scored for question-answer relevance, the top 10K become positive examples, and 10K negative examples are created by randomly pairing questions with other answers. A binary classifier is trained by embedding questions and answers with sentence-transformers/all-MiniLM-L12-v2 and minimizing cosine distance. A threshold of 0.1 is manually determined from a plot of embedding scores. Questions with average answer-embedding score below 0.1 are removed. The final dataset is 11.4M questions with 10B+ tokens.

• ArXiv: The ArXiv subset of RedPajama, further processed to retain only LaTeX source files and remove preambles, comments, macros, and bibliographies, resulting in approximately 30B tokens.

• Wikipedia: English Wikipedia from RedPajama (2023-03-20 dump), with hyperlinks and templates removed, totaling approximately 6B tokens.

• OpenWebMath: High-quality mathematical text from CommonCrawl, approximately 15B tokens.

Preprocessing Pipeline: Deduplication, Privacy, Safety, and Opt-Outs

Not all preprocessing steps are applied to all data sources. Table 3 provides a matrix: source code, pull requests, notebooks, and issues receive all five preprocessing steps (deduplication, malicious code removal, decontamination, opt-out removal, and PII redaction). Documentation and LHQ receive only a subset. The specific mechanisms are:

Near-Deduplication (Section 3.1): Applied to source code, pull requests, notebooks, issues, and documentation. The pipeline follows SantaCoder's approach: (1) compute MinHashes of all files using 5-grams; (2) use Locality Sensitive Hashing (LSH) to group similar files; (3) retain only one file from each duplicate group. The Jaccard similarity threshold is 0.7. To preserve repository context, when choosing which file to retain from a duplicate group, files from repositories with higher star/fork counts or from the latest commit date are prioritized. Natural language datasets (ArXiv, StackExchange, OpenWebMath, Wikipedia, LHQ) are not deduplicated because they are already preprocessed by their original distributors.

PII Redaction (Section 3.2): The StarPII model (from StarCoder) is used to redact names, emails, keys, passwords, IP addresses, and usernames from source code, pull requests, issues, and StackOverflow. For OpenWebMath and documentation, only names, keys, and emails are redacted. For ArXiv, only emails are redacted using a regex from Ben Allal et al. (2023).

Username anonymization in conversations (issues, PRs, StackOverflow) works by substituting author usernames with a participant counter within the conversation (e.g., username_1 for the second participant). These pseudonyms are placed at the start of each comment. References to these usernames within message text are removed. Only actively participating usernames are masked; mentions of non-participating users remain unaffected.

Decontamination (Section 3.3): To prevent benchmark inflation, the training set is scrubbed of files containing docstrings or solutions from HumanEval and MBPP, docstrings from APPS, questions from GSM8K, and prompts from DS-1000. The paper notes an enhancement over StarCoder1: "we further enhance the recall of the decontamination process by removing whitespace during string matching." This addresses the weakness that simple string matching can miss near-identical code snippets where only whitespace differs. Documentation, LHQ, ArXiv, and Wikipedia are not decontaminated.

Malware Removal (Section 3.4): ClamAV 1.2 with additional unofficial signatures from SaneSecurity (as of 2023-11-16) is used to scan source code, pull requests, notebooks, and issues. Signatures with high false-positive risk (as determined by SaneSecurity) are excluded. This eliminates 59,442 files (0.009% of 654M total files). The top detected signature is Sanesecurity.Malware.28845.BadVBS with 11,876 hits. The top language by potentially malicious file count is Text (13,281 files), followed by HTML (11,336) and JavaScript (10,210).

Opt-Out Removal (Section 3.5): The "Am I in The Stack" tool was updated with The Stack v2 repositories. A public announcement on X (Twitter) on November 20, 2023 set a cut-off date for opt-out requests. After this date, 1,561 repositories from 91 users and organizations were removed, totaling 22,066 files from the source code dataset (excluding issues and PRs, which were not part of the opt-out scope).

Data Composition: Who Gets What and Why

The paper makes a deliberate choice to train different model sizes on different data compositions, motivated by the principle that smaller models have limited capacity and benefit from less diverse data (Section 4). This is supported by multilingual NLP research showing languages compete for model capacity (Arivazhagan et al., 2019; Conneau et al., 2020; Scao et al., 2022b).

The-stack-v2-train-smol (525B tokens): Used for 3B and 7B models. Contains 17 programming languages: C, C#, C++, Go, Java, JavaScript, Kotlin, Lua, PHP, Python, R, Ruby, Rust, SQL, Shell, Swift, TypeScript. Also includes documentation-related languages (AsciiDoc, HTML, Markdown, RDoc, RMarkdown, Text, reStructuredText) and configuration languages/files (Ant Build System, CMake, Dockerfile, Go Module, Gradle, INI, Java Properties, Makefile, Maven POM, TOML, plus a list of specific config filenames in Appendix C.1).

The-stack-v2-train-full (775B tokens): Used only for the 15B model. Includes all 619 programming languages. Adding 600+ languages beyond the smol subset contributes only approximately 250B additional tokens.

Downsampling: Before constructing the datasets, the paper examines the natural language distribution and finds significant skew: Java (479.68 GB), JavaScript (277.25 GB), C++ (204.49 GB), Python (190.99 GB), PHP (171.57 GB), C# (166.22 GB), C (114.49 GB). Both Java and JavaScript are downsampled to 200 GB to put high-resource languages on a more equal footing. Markdown is preserved at 254 GB (anticipated to contain more code documentation), while HTML is reduced to 100 GB (associated with webpages). Data formats like JSON, XML, and YAML are subsampled to 8 GB, and several other languages are subsampled to 1 GB (full list in Table 28).

Data composition per model size (Table 4):

The 3B model excludes OpenWebMath, Wikipedia, and ArXiv to reduce diversity for the smallest model. The 7B model includes these natural language datasets but uses the smol source code subset. The 15B model gets everything, totaling 913B+ unique tokens—4× larger than StarCoderBase's training set.

Repository context: After subsampling, source code files from Software Heritage are compiled into repository-context-aware datasets. Each example is a full repository with files arranged in random order within the same repository. This is a change from StarCoder1's file-context approach where random files from different repositories were concatenated together.

Data Formatting: How Raw Text Becomes Model Input

Each data source is formatted with specialized sentinel tokens that delineate its structure. These sentinel tokens (Table 5) serve as structured metadata that the model learns to interpret during pretraining.

Source Code Formatting (Section 5.1): Two variants are used, with the choice made randomly at 50% probability:

With repository metadata: <repo_name>reponame<file_sep>filepath1\ncode1<file_sep>filepath2\ncode2 ... <|endoftext|>

Without repository metadata: <file_sep>code1<file_sep>code2 ... <|endoftext|>

The 50% probability for metadata inclusion enables the model to operate both with and without file path context—important for use cases where repository structure is unknown.

Fill-in-the-Middle (FIM) transformation: Applied to source code at the repository level using a two-stage probabilistic scheme. First, repositories are selected for FIM with 50% probability. Within a selected repository, files are split by <|endoftext|> and <file_sep> tokens into chunks, and each chunk independently receives FIM transformation with 50% probability. Repository metadata (<repo_name>reponame) is never FIM-transformed.

The FIM format for a chunk uses three sentinel tokens: <fim_prefix>, <fim_suffix>, and <fim_middle>. A code chunk is split at a random position into prefix, middle, and suffix parts. The model sees <fim_prefix>prefix<fim_suffix>suffix<fim_middle> during training and must predict the middle part conditioned on both prefix and suffix. This enables code infilling—completing code given surrounding context—without requiring a separate infilling-specific model.

Example of FIM applied to the second file in a repository:

<repo_name>reponame<file_sep>filepath0\ncode0<file_sep><fim_prefix>filepath1\ncode1_pre<fim_suffix>code1_suf<fim_middle>code1_mid<file_sep> ...<|endoftext|>

Pull Request Formatting (Section 5.2): This is the most complex formatting scheme. PRs are rendered into structured sequences with multiple temporal blocks.

For base files: with 0.2 probability, the entire file is included; otherwise, only a range of changes across all commit heads is shown, with up to 32 randomly sampled context lines before and after the changes.

Diff hunks display modifications between before/after states, with 3–10 randomly selected context lines on each side.

The format has two repeating blocks:

Block 1 (initial state):

<pr>Title: title\nusername_0: description
<pr_status>opened
<repo_name>reponame
<pr_base>
<pr_file>filepath_1
<pr_base_code>file_content/changes_1
...
<pr_file>filepath_N
<pr_base_code>file_content/changes_N
<pr_diff>
<pr_file>filepath_1
<pr_diff_hunk>diff_hunk_1
...
<pr_diff_hunk>diff_hunk_K
...
<pr_file>filepath_M
<pr_diff_hunk>diff_hunk_1
...
<pr_diff_hunk>diff_hunk_J

Block 2 (repeated per head commit): Contains general comments (<pr_comment>), review comments (<pr_review> with <pr_review_state> indicating approved/rejected/commented/changes_required), and code review comments (<pr_review_comment> with file path, line number reference, and diff hunk). Each comment includes an event ID for threading. The block concludes with diff hunks between the PR base and the new head, reflecting the outcome of discussions.

Final block (when PR is closed):

<pr>username_id
<pr_status>closed
<pr_is_merged>True
<|endoftext|>

Why this format matters: The temporal sequencing of base files → initial diffs → review comments → revised diffs → merge decision teaches the model the structure of code review. Comments are threaded via <pr_in_reply_to_review_id> and <pr_in_reply_to_comment_id>, preserving conversational structure. The inclusion of review states (approved, rejected, changes_required, commented) provides supervision signals about code quality. When a PR is merged (<pr_is_merged>True), the model learns that the final code changes represent an accepted solution to the described problem.

GitHub Issues Formatting (Section 5.3):

<issue_start>Title: title\nusername_id0: comment_0<issue_comment>username_id1: comment_1
... <issue_closed (optional)><issue_comment>username_idn: comment_n<|endoftext|>

Comments are separated by <issue_comment> tokens. Anonymized speaker identifiers precede each comment. The <issue_closed> token is added if the issue was closed. This format is later repurposed at evaluation time for bug-fixing and code editing tasks (Sections 7.2.1 and 7.2.2).

Jupyter Notebooks Formatting (Section 5.4):

Scripts: <jupyter_script>code<|endoftext|>

Structured:

<jupyter_start><jupyter_text>text0<jupyter_code>code0<jupyter_output>output0<jupyter_text> ... <|endoftext|>

The <empty_output> token is used when an output cell is empty.

Kaggle scripts (with schema information when available):

<jupyter_start><jupyter_text>title\ndescription\nKaggle dataset identifier: data_identifier
<jupyter_code>import pandas as pd\n\ndf = pd.read_csv(data_path1)\ndf.info()
<jupyter_output>df_info_output1
<jupyter_text>Examples:\nexample1_1\n..example1_4
...
<jupyter_script>code<|endoftext|>

The <jupyter_script> token distinguishes the full notebook script (which can be long) from short code snippets following <jupyter_code>.

Kaggle structured: Same as Jupyter structured but without output cells: <jupyter_start><jupyter_text>text0<jupyter_code>code0<jupyter_text> ... <|endoftext|>

StackExchange Formatting (Section 5.5):

<issue_start>username_id0: question
<issue_comment>username_id1: answer_1\nUpvotes: score [selected answer](Optional)
...
<issue_comment>username_idn: answer_n\nUpvotes: score [selected answer](Optional)<|endoftext|>

Answers are in random order. Upvote scores and selected-answer status are included as metadata. This format reuses the issue sentinel tokens, making StackOverflow discussions structurally similar to GitHub issues—the model learns a shared representation for technical Q&A regardless of platform.

Intermediate Representations Formatting (Section 5.6):

code<code_to_intermediate>intermediate_representation
intermediate_representation<intermediate_to_code>code

A 50/50 split between source→IR and IR→source directions. Size-optimized IR is used 80% of the time, performance-optimized 20%.

Model Architecture and Training: The Transformer Details

Architecture (Section 6.1, Table 6): StarCoder2 uses decoder-only transformer architectures with two key modifications relative to StarCoderBase:

1. Rotary Positional Encodings (RoPE): Replaces learned positional embeddings with RoPE (Su et al., 2021), with a base period of θ = 1e5 for 3B and 7B models, and θ = 1e4 for 15B (the latter is described as a configuration parsing bug). The paper states this was motivated by "significant performance gains in a preliminary ablation study."

2. Grouped Query Attention (GQA): Replaces Multi-Query Attention (Shazeer, 2019) with GQA (Ainslie et al., 2023). The number of key-value heads is kept low to avoid slowing down inference: 2 for 3B, 4 for 7B and 15B.

FLOPs are estimated with the standard approximation:

$X = 6ND$

where $N$ is the number of parameters and $D$ is the number of training tokens. The estimate includes both base and long-context training.

Tokenizer (Section 6.2): A byte-level Byte-Pair-Encoding (BPE) tokenizer is trained on a small subset of The Stack v1 with vocabulary size 49,152 tokens (including the sentinel tokens from Table 5). The pre-tokenization step uses a digit splitter and the GPT-2 regex splitter. The paper notes that "increasing the vocabulary size to 100K did not improve performance" in preliminary experiments, so the vocabulary size is unchanged from StarCoderBase.

Base Model Training (Section 6.3, Table 7):

All models are trained with:

• Optimizer: Adam (Kingma & Ba, 2015) with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$
• Weight decay: 0.1
• No dropout
• Learning rate schedule: cosine decay after 1,000 iterations of linear warmup
• Sequence length: 4,096 tokens

The 15B model was scheduled for 1.1M iterations but was early-stopped after 1M. The RoPE θ of 1e4 for 15B (versus 1e5 for 3B and 7B) is attributed to a configuration parsing bug.

Why train 4–5 epochs? The paper explicitly states they "push the number of training tokens far beyond the compute-optimal number suggested by Chinchilla (Harm's law; de Vries, 2023)"—a Chinchilla-optimal 3B model would train on approximately 60B tokens, but StarCoder2-3B trains on 3.1T tokens (~52× Chinchilla-optimal). This is aligned with the finding of Muennighoff et al. (2023) that in data-constrained regimes, repeating data for multiple epochs can yield continued improvements. The dataset contains 622–913B unique tokens, but the models see 3.1–4.1T tokens through repetition.

Long Context Fine-Tuning (Section 6.3, Table 8):

After base training, each model is further trained for 200B tokens at 16,384 context length using FlashAttention-2 (Dao, 2024) with a sliding window of 4,096.

RoPE θ is increased for long-context training (to 1e6 for 3B and 7B, to 1e5 for 15B) to accommodate the extended sequence length.

CO₂ Emissions (Section 6.4):

• 3B: 97,120 GPU-hours on A100 SXM4 80GB (TDP 400W) at ServiceNow (carbon efficiency 0.386 kgCO₂eq/kWh). Base: 14,995.33 kgCO₂eq; long-context: 1,111.68 kgCO₂eq; total: 16,107.01 kgCO₂eq.
• 7B: 145,152 GPU-hours on H100 (TDP 660W) at Hugging Face (carbon efficiency 0.2925 kgCO₂eq/kWh). Base: 28,021.6 kgCO₂eq; long-context: 1,601.23 kgCO₂eq; total: 29,622.83 kgCO₂eq.
• 15B: Estimate not yet provided at time of publication.

Evaluation Methodology: How Performance Is Measured

The evaluation suite spans seven categories, each with specific hyperparameters. I'll explain the reasoning behind key choices.

Code Completion Benchmarks:

HumanEval/MBPP with EvalPlus (Section 7.1.1): These are the standard code generation benchmarks where models complete a function given a docstring/signature (HumanEval) or natural language description with tests (MBPP). EvalPlus (Liu et al., 2023a) augments the original benchmarks with 80× (HumanEval+) and 35× (MBPP+) more test cases to catch subtle bugs that pass the original weak test suites. All models use greedy decoding and report mean pass@1. EvalPlus uses 399/427 MBPP problems after omitting ill-formed ones.

MultiPL-E (Section 7.1.2): Translates HumanEval to 18 other programming languages via rule-based compilers, testing multilingual code generation. Models sample 50 completions per prompt at temperature 0.2, top-p 0.95, reporting pass@1.

DS-1000 (Section 7.1.3): 1,000 data science tasks across 7 Python libraries (Matplotlib, NumPy, Pandas, PyTorch, SciPy, Scikit-Learn, TensorFlow). Models generate 40 samples per problem at temperature 0.2, top-p 0.95, reporting mean pass@1.

RepoBench (Section 7.6.1): Repository-level next-line prediction. Three settings (cross-file-first, cross-file-random, in-file) at 5 context lengths (2K, 4K, 8K, 12K, 16K). Models generate at temperature 0.2, top-p 0.95, with the first non-empty, non-comment line taken as prediction. Max cross-file context is truncated to 15,800 tokens (7,800 for StarCoderBase due to its 8K limit).

CrossCodeEval (Section 7.6.2): Repository-level completion requiring cross-file context. Uses retrieve-and-generate with OpenAI's ada embedding, retrieving up to 100 code segments (each with file path and 10 lines of code). Max cross-file context is 12,800 tokens; max generation is 50 tokens at temperature 0.2, top-p 0.95.

Code Fixing and Editing:

HumanEvalFix (Section 7.2.1): Bug-fixing across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Models are evaluated with both standard "Instruct" prompts and the custom "Issue" prompt that mimics GitHub issue discussions:

<issue_start>username_0: instruction\n\n```buggy function```\nUpvotes: 100<issue_comment>username_1: Sure, here is the fixed code.\n\n```function start

Generation stops when ``` is produced. Temperature 0.2, 20 samples per problem. The paper evaluates instruction-tuned models (DeepSeekCoder-Instruct, CodeLlama-Instruct, OctoCoder) alongside base StarCoder2 with the Issue prompt.

CanItEdit (Section 7.2.2): Python code editing with descriptive and lazy instructions. Temperature 0.2, top-p 0.95, 100 completions per problem. Uses the Issue prompt format for StarCoder2.

Math and Code Reasoning:

GSM8K with PAL (Section 7.3): 8-shot evaluation with greedy decoding. Models generate a Python program (Program-Aided Language model approach) which is executed to produce the answer.

CRUXEval (Section 7.4): 800 samples testing code execution reasoning. Two sub-tasks: CRUXEval-I (predict input given function and output) and CRUXEval-O (predict output given function and input). Temperature 0.2 for pass@1 and temperature 0.8 for pass@5, both with 10 samples.

Fill-in-the-Middle (Section 7.5): Tests the ability to fill a single line of code in HumanEval solutions across Java, JavaScript, and Python. 20 completions per example at temperature 0.2, top-p 0.95, reporting exact match.

Security (Section 7.7): "Asleep at the Keyboard" benchmark with 18 vulnerability classes from MITRE CWE Top 25 (2021). 23 C scenarios and 17 Python scenarios, 25 samples per scenario (1,000 total) at temperature 0.2, top-p 0.95.

Bias and Toxicity (Section 7.8): BOLD (23,679 prompts across 5 dimensions, 1 completion per prompt, up to 50 tokens), WinoBias (perplexity-based gender bias measurement), HONEST (5 completions per sample, up to 50 tokens), RealToxicityPrompts (10,000-subset, 1 completion per prompt, up to 50 tokens).

4. Key Insights and Innovations

Innovation 1: Full Training Data Transparency as a Competitive Advantage, Not a Concession

The dominant narrative in Code LLM development has been that releasing training data is a tradeoff—a sacrifice of competitive advantage in exchange for ethical principle. Proprietary models (GPT-4, Gemini) and open-weight-but-closed-data models (Code Llama, DeepSeekCoder) implicitly argue that opacity protects business interests or prevents misuse. StarCoder2 challenges this framing by demonstrating that full data transparency is not merely compatible with state-of-the-art performance; it is the engine of specific capabilities that closed-data models struggle to match.

The paper's most intellectually distinctive move is treating training data as a first-class scientific artifact rather than a trade secret. This isn't just about releasing a data dump—it's about building an auditable, governable data supply chain. The Software Heritage persistent identifiers (SWHIDs) assigned to every source code component mean that any file in the training set can be traced to its exact version in a specific repository at a specific commit. This transforms training data from an opaque blob into a versioned, attributable, queryable knowledge base (Section 9.2). When the paper claims StarCoder2-15B outperforms DeepSeekCoder-33B on low-resource languages like D and Perl (Table 10), a researcher using StarCoder2 can investigate why by tracing those language-specific files through the SWHID system to understand their provenance, quality, and characteristics. With DeepSeekCoder, this investigation is impossible.

This reframing has concrete implications beyond ideology. The paper's own decontamination process (Section 3.3) acknowledges that prior methods were insufficient—whitespace-insensitive matching represents a genuine improvement in benchmark integrity. But this improvement is only verifiable because the training data is public. Any external researcher can audit the decontamination claims, run their own contamination analyses, and trust (or challenge) the reported benchmark scores. For closed-data models, benchmark results operate on faith. StarCoder2 demonstrates that openness and rigor are mutually reinforcing: the need to publicly justify data curation decisions drives more careful engineering.

The governance infrastructure—the opt-out tool, the Bloom filter membership checker (Section 8), the Elasticsearch index—is not a side project but a central contribution. It operationalizes the idea that training data transparency requires interactive, queryable tools, not just static documentation. The "Data Portrait" Bloom filter (Section 8) enables content creators to check whether their work appears verbatim in the dataset without downloading terabytes of data—a lightweight membership testing mechanism that Marone & Van Durme (2023) argued should be standard practice but which had not been implemented at this scale for code data before. This is incremental relative to StarCoder1's governance tools, but the expansion to non-code sources (documentation, textbooks, papers) and the implementation of a separate Bloom filter for prose data sources represent a significant scaling of the governance approach to heterogeneous training data.

The evidence that transparency produces competitive results is distributed across the evaluation section: StarCoder2-15B matches or exceeds CodeLlama-34B on 10/18 MultiPL-E languages (Table 10), outperforms DeepSeekCoder-33B on GSM8K by 6.4 percentage points (Table 14), and surpasses both on CRUXEval-I (Table 15). These results establish that the open-data model family is not sacrificing performance for principle—it is outperforming opaque models on specific, practically important dimensions (math reasoning, low-resource languages, code execution understanding) while maintaining full auditability.

Innovation 2: Instruction-Following Capability Through Pretraining Data Design Rather Than Post-Hoc Fine-Tuning

The standard recipe for making LLMs follow instructions involves a two-stage pipeline: pretrain a base model on raw text, then fine-tune it on curated instruction-response pairs (Wei et al., 2022; Sanh et al., 2022). This approach has been widely adopted for Code LLMs, with models like OctoCoder (Muennighoff et al., 2024a) and DeepSeekCoder-Instruct (Guo et al., 2024) relying on explicit instruction tuning to enable code editing and bug-fixing capabilities. The underlying assumption is that base models pretrained on source code cannot effectively follow natural language editing instructions without this additional fine-tuning stage.

StarCoder2 challenges this assumption through a data design strategy that embeds instruction-following structure directly into pretraining. The pull request format (Section 5.2) is not merely a way to include more data—it is a carefully constructed simulation of the instruction-following paradigm within the pretraining distribution. Each PR represents a natural instruction ("here is the proposed change"), a response (the diff), feedback (review comments), and revision (updated diffs), all annotated with outcomes (approved, rejected, changes_required, merged). The model learns to map between natural language instructions and code modifications not because it was explicitly fine-tuned to do so, but because this mapping is a fundamental structure in its pretraining data.

The evidence for this claim is the dramatic gap between the standard "Instruct" prompt and the "Issue" prompt on HumanEvalFix (Table 12). StarCoder2-15B as a base model achieves only 22.0% average pass@1 with a standard instruct prompt, but 38.7% with the Issue prompt—a 76% relative improvement. This is not merely prompt engineering; it demonstrates that the model has internalized the discourse structure of code review discussions from pretraining and can leverage that structure at inference time without any parameter updates. The Issue prompt works because it activates a schema the model already knows: <issue_start> → problem description → <issue_comment> → proposed solution. The model learned during pretraining that sequences following this pattern involve code fixes, and it reproduces that behavior.

This finding reframes the relationship between pretraining and instruction tuning. If instruction-following behavior can be taught through careful pretraining data design, then instruction tuning becomes a matter of alignment and formatting convenience rather than a fundamental capability injection. The paper acknowledges that instruction tuning would still be valuable (Section 7.2.1 notes that "additional prompt engineering may be necessary" and that instruction tuning would "further improve its usability"), but the core capability—understanding a natural language description of a code change and producing the modified code—is already present. This is a fundamental shift from the "pretrain then instruct-tune" paradigm toward a "design pretraining data to include natural instruction-response pairs" paradigm.

The CanItEdit results (Table 13) reinforce this interpretation: StarCoder2-15B with the Issue prompt achieves 43.08% on descriptive instructions and 38.45% on lazy instructions, outperforming CodeLlama-Instruct-34B (35.0% and 26.76% respectively) despite the latter being both larger and explicitly instruction-tuned. The base model, through pretraining alone, has acquired more robust code editing capabilities than a larger model that required a separate fine-tuning stage. This suggests that the pretraining-data-as-instruction-data approach may be not just competitive with explicit instruction tuning but potentially more sample-efficient in terms of total training compute.

Innovation 3: Repository-Level Context as a First-Class Training Objective, Not an Inference-Time Hack

Prior Code LLMs largely treated code files as independent documents. Repository-level code completion—where understanding cross-file dependencies is necessary—was typically addressed at inference time through retrieval mechanisms: identify relevant files from the repository, stuff them into the context window, and hope the model can use them (Ding et al., 2023; Liu et al., 2023b). StarCoderBase trained with file-level context, meaning the model saw random files concatenated together without any signal about which files belonged to the same project. This approach works for isolated function completion but fundamentally fails to capture the relational structure of software—imports, class hierarchies, configuration files, build scripts, and module dependencies that span multiple files.

StarCoder2 makes repository context a first-class training objective through a simple but consequential formatting choice: grouping files from the same repository together in random order, with file paths included as metadata, and applying FIM transformations at the repository level (Section 5.1). This teaches the model during pretraining—not at inference time—that code files are not independent; they exist within a project structure where the contents of one file constrain and inform the contents of others.

What makes this conceptually distinctive is the recognition that the training data format is itself a form of curriculum design. By presenting files from the same repository together, the model sees co-occurrence patterns that reflect genuine software engineering dependencies: a Python file that imports from another file in the same repo, a C header file and its implementation, a package.json alongside the JavaScript files it configures, a CMakeLists.txt with the C++ files it builds. The 50% probability of including file paths (Section 5.1) is a deliberate design choice that balances two objectives: teaching the model to use path information when available while maintaining the ability to operate without it (important for single-file code completion scenarios).

The evaluation evidence for this innovation is the consistent improvement of StarCoder2 over StarCoderBase on repository-level benchmarks (Tables 17 and 18). On RepoBench (Table 17), StarCoder2-3B improves over StarCoderBase-3B by 2.48 percentage points in Python exact match and 2.45 points in Java exact match; StarCoder2-15B improves over StarCoderBase-15B by 3.48 points in Python and 3.23 points in Java. On CrossCodeEval (Table 18), the improvements are even more striking: StarCoder2-15B achieves 78.72% Python Code Match versus StarCoderBase-15B's 73.43%—a 5.29 point improvement. These gains cannot be attributed to model size (same architecture as competitors), training data volume (StarCoderBase already had substantial data), or inference-time retrieval (both models use the same retrieve-and-generate setup). The difference is that StarCoder2 learned during pretraining that code files have inter-file relationships, while StarCoderBase treated all code as context-free text.

The limitation is that StarCoder2-15B still underperforms on TypeScript in CrossCodeEval (48.63% vs. 68.98% for CodeLlama-34B, Table 18) and does not dominate all repository-level benchmarks. The random file ordering within repositories is a crude approximation of actual project structure—real repositories have meaningful directory hierarchies, conventions (e.g., src/, tests/, docs/), and import graphs that random ordering destroys. This suggests that repository context as a pretraining objective is not yet solved; the paper's approach demonstrates the importance of the concept but leaves substantial room for more sophisticated repository structure modeling (e.g., topological ordering by dependency graph, explicit encoding of import relationships).

Innovation 4: Over-Training Small Models on Massive Code Data as a Viable Scaling Strategy

The Chinchilla scaling laws (Hoffmann et al., 2022) established a compute-optimal relationship between model size and training tokens: for a model with $N$ parameters, the optimal number of training tokens is approximately $20N$. This framework has guided most LLM training decisions, with the implicit assumption that deviating from the optimal ratio wastes compute—training on more tokens yields diminishing returns, and training on fewer tokens leaves model capacity underutilized.

StarCoder2 systematically violates this principle at all three model sizes, and the results challenge the universality of Chinchilla-optimal scaling for code-specific models. The 3B model trains on 3.1T tokens—approximately 52× the Chinchilla-optimal 60B tokens. The 7B model trains on 3.5T tokens (25× Chinchilla-optimal). The 15B model trains on 4.1T tokens (14× Chinchilla-optimal). All three models see 4–5 epochs over their data, despite the conventional wisdom that repeating data is less efficient than training on unique tokens (Muennighoff et al., 2023).

The paper frames this as training "far beyond the compute-optimal number suggested by Chinchilla (Harm's law; de Vries, 2023)" (Section 1), but the results suggest this is not merely a brute-force resource expenditure—it is a strategic choice enabled by the unique characteristics of code data. Code, unlike general web text, has a highly structured, repetitive, and compositional nature. The same programming constructs (loops, conditionals, function definitions) appear across millions of files with minor variations. Deep learning on code, the paper's results imply, behaves differently from deep learning on natural language: the model continues to benefit from additional exposures to the same syntactic patterns and idiomatic structures even after the "optimal" number of unique tokens has been consumed.

The evidence that this strategy works is the performance of StarCoder2-3B, which "matches or surpasses the performance of StarCoderBase-15B" (Section 1) despite having 5× fewer parameters. Specifically, StarCoder2-3B scores higher than StarCoderBase-15B on HumanEval (31.7 vs. 29.3), MBPP (57.4 vs. 50.6), DS-1000 (25.0 vs. 23.8—though StarCoderBase-15B's score is from Table 11, where it achieves 23.8 overall), and most MultiPL-E languages (Table 10). This is not merely a data quality story; the data curation improvements (unlicensed code, PRs, documentation) cannot alone explain why a 3B model surpasses a 15B model trained in a similar paradigm. The 5× training budget advantage (3.1T vs. ~600B tokens for StarCoderBase-15B, though exact StarCoderBase token counts are not stated in this paper) combined with data repetition appears to be a significant factor.

However, the 7B model complicates this narrative. Trained on 3.5T tokens (5.3 epochs), it "does not perform as well as StarCoder2-3B and StarCoder2-15B for their size" (Section 1). This is an important negative result that the paper acknowledges but does not explain. If over-training were uniformly beneficial, the 7B model—trained on more tokens than the 3B with a richer data mixture (including OpenWebMath, Wikipedia, ArXiv)—should outperform the 3B by a margin consistent with its 2.3× parameter advantage. Instead, it lags behind DeepSeekCoder-6.7B substantially (Table 10 shows DeepSeekCoder-6.7B outperforming StarCoder2-7B on 14/18 MultiPL-E languages). This suggests that over-training interacts non-monotonically with model capacity and data diversity: the 7B model may be in a regime where the additional natural language data dilutes code-specific performance without sufficient model capacity to integrate both modalities effectively, while the 3B model's restricted data mixture (code-only without natural language) avoids this interference. This is a diagnostic negative result that reveals an unresolved tension between data diversity and model capacity in the multi-epoch training regime—a finding that should inform future scaling decisions but which the paper leaves as an open puzzle.

The conceptual contribution is not "over-training is good" but rather that the Chinchilla framework, developed for general-domain language modeling, does not cleanly transfer to code—and that understanding this failure mode requires distinguishing between data diversity (adding new domains/languages) and data repetition (re-exposing the model to the same distribution). The 3B model succeeds with extreme over-training on a narrow distribution; the 15B model succeeds with moderate over-training on a broad distribution; the 7B model falls in an intermediate regime where the combination of moderate over-training and intermediate diversity appears suboptimal. This is a more nuanced finding than simply reporting that over-training works or that more data is better.

5. Experimental Analysis

Evaluation Methodology

• Dataset(s). The evaluation spans numerous benchmarks. For code completion, the primary datasets are HumanEval (Chen et al., 2021) with 164 problems and MBPP (Austin et al., 2021) with 399 sanitized problems, both augmented by EvalPlus (Liu et al., 2023a) to provide 80× and 35× more tests respectively. MultiPL-E (Cassano et al., 2023b) extends HumanEval to 18 additional programming languages. DS-1000 (Lai et al., 2023) provides 1,000 data science tasks across seven Python libraries. RepoBench v1.1 (Liu et al., 2023b) and CrossCodeEval (Ding et al., 2023) test repository-level code completion. HumanEvalFix (Muennighoff et al., 2024a) tests bug-fixing across six languages, and CanItEdit (Cassano et al., 2024) evaluates code editing with descriptive and lazy instructions. GSM8K (Cobbe et al., 2021) with PAL (Gao et al., 2023) tests math reasoning. CRUXEval (Gu et al., 2024) with 800 samples tests code execution reasoning. The "Asleep at the Keyboard" benchmark (Pearce et al., 2022) evaluates security vulnerabilities. BOLD (Dhamala et al., 2021), WinoBias (Zhao et al., 2018), HONEST (Nozza et al., 2021), and RealToxicityPrompts (Gehman et al., 2020) measure bias and toxicity.

• Base model(s). The paper evaluates three StarCoder2 models (3B, 7B, 15B parameters) trained as described in Section 6. These are base (non-instruction-tuned) models. For comparison, the paper evaluates against base models from StableCode 3B (Pinnaparaju et al., 2024), Code Llama 7B and 13B (Rozière et al., 2023), DeepSeekCoder 1.3B, 6.7B, and 33B (Guo et al., 2024), StarCoderBase 3B, 7B, and 15B (Li et al., 2023), and the larger CodeLlama-34B. On instruction-following benchmarks (HumanEvalFix, CanItEdit), instruction-tuned variants of these models are used as baselines alongside StarCoder2 base models prompted in the custom "Issue" format.

• Metrics. The primary metric across code completion benchmarks is pass@1, computed either with greedy decoding (HumanEval, MBPP) or by sampling $k$ completions at temperature 0.2 with top-p 0.95 and reporting the fraction of problems solved at least once, as specified per benchmark. MultiPL-E reports pass@1 averaged over 50 samples per problem. DS-1000 reports mean pass@1 over 40 samples. RepoBench reports edit similarity, exact match, and CodeBLEU. CrossCodeEval reports Code Match (edit similarity) and Identifier Match (F1 score). HumanEvalFix and CanItEdit report pass@1 at temperature 0.2. GSM8K reports 8-shot accuracy with greedy decoding. CRUXEval reports pass@1 (temperature 0.2) and pass@5 (temperature 0.8) with 10 samples each. The security benchmark reports the percentage of valid completions and the percentage of valid completions that contain the tested vulnerability. Bias benchmarks report classifier scores: Regard scores for BOLD, perplexity-based bias for WinoBias, HONEST hurtfulness scores, and a toxicity probability for RealToxicityPrompts.

• Baselines. The paper compares against: StarCoderBase at 3B, 7B, and 15B (Li et al., 2023)—the direct predecessor within the BigCode project family; StableCode-3B (Pinnaparaju et al., 2024)—a contemporaneous small Code LLM; CodeLlama at 7B, 13B, and 34B (Rozière et al., 2023)—the dominant open-weight Code LLMs at release; DeepSeekCoder at 1.3B, 6.7B, and 33B (Guo et al., 2024)—the strongest open-weight models on many code completion benchmarks. For instruction-following benchmarks, OctoCoder-15B (Muennighoff et al., 2024a)—an instruction-tuned StarCoderBase variant using CommitPackFT—and instruction-tuned variants of CodeLlama and DeepSeekCoder are included.

• Generation budget / compute accounting. The paper does not use a unified test-time compute budget paradigm. Instead, each benchmark specifies its own sampling hyperparameters (temperature, top-p, number of completions) consistent with the conventions established by the respective benchmark creators. The evaluation is therefore a set of standardized assessments under fixed inference protocols, not a compute-optimal scaling analysis. There is no FLOPs counting or budget constraint for inference—the goal is to measure capability under consistent conditions, not to optimize inference-time resource allocation.

• Cross-validation / statistical protocol. Some benchmarks report inherent noise characteristics: for CRUXEval, the original paper reports approximately 0.2% noise from 10-sample sampling and approximately 1.5% noise from the selection of 800 benchmark examples from a larger pool. For MultiPL-E and DS-1000, multiple samples (50 and 40 respectively) are drawn per problem to estimate pass@1, following standard practice. HumanEval and MBPP use greedy decoding, producing deterministic scores. The paper does not report confidence intervals, standard deviations, or statistical significance tests for most benchmarks. The RepoBench and CrossCodeEval results are reported as raw metric values without uncertainty quantification.

Main Quantitative Results

Code Completion: HumanEval, MBPP, and EvalPlus (Section 7.1.1)

The headline finding is that StarCoder2-3B is the best-performing small model on all four benchmarks, and StarCoder2-15B is the best-performing large model by a significant margin. Table 9 reports greedy-decoding pass@1.

For the small models (≤3B parameters), StarCoder2-3B achieves 31.7 on HumanEval, 57.4 on MBPP, 27.4 on HumanEval+, and 47.4 on MBPP+. This represents a 60.2% relative improvement over StarCoderBase-3B on HumanEval+ (17.1 → 27.4) and a 32.4% improvement on MBPP+ (35.8 → 47.4). Against StableCode-3B, the gains are smaller but consistent: 28.7 → 31.7 on HumanEval, 24.4 → 27.4 on HumanEval+. Against DeepSeekCoder-1.3B, the improvement is 28.7 → 31.7 on HumanEval and 23.8 → 27.4 on HumanEval+.

For the medium models (≤7B), StarCoder2-7B achieves 35.4 on HumanEval and 54.4 on MBPP. This places it second behind DeepSeekCoder-6.7B, which scores 47.6 on HumanEval and 70.2 on MBPP. The gap to DeepSeekCoder-6.7B is substantial: the paper reports DeepSeekCoder-6.7B outperforms StarCoder2-7B by 32.4% on HumanEval+ (39.6 vs. 29.9) and 24.1% on MBPP+ (56.6 vs. 45.6). However, StarCoder2-7B consistently outperforms CodeLlama-7B (35.4 vs. 33.5 on HumanEval; 29.9 vs. 25.6 on HumanEval+) and StarCoderBase-7B (19.6% improvement on HumanEval+).

For the large models (≤15B), StarCoder2-15B achieves 46.3 on HumanEval, 66.2 on MBPP, 37.8 on HumanEval+, and 53.1 on MBPP+. Against CodeLlama-13B, this represents a 22.5% relative improvement on HumanEval (37.8 → 46.3) and a 17.0% improvement on HumanEval+ (32.3 → 37.8). Against StarCoderBase-15B, the improvements are dramatic: 47.7% relative improvement on HumanEval+ (25.6 → 37.8). StarCoder2-15B outperforms CodeLlama-34B on MBPP (66.2 vs. 65.4) and MBPP+ (53.1 vs. 52.4), though it trails on HumanEval (46.3 vs. 48.2). DeepSeekCoder-33B remains the strongest overall on these benchmarks, scoring 54.3 on HumanEval and 73.2 on MBPP.

Multilingual Code Completion: MultiPL-E (Section 7.1.2)

Table 10 reports pass@1 across 18 programming languages sampled at temperature 0.2 with top-p 0.95 and 50 samples per problem. The headline: StarCoder2-3B is the best small model on 11/18 languages; StarCoder2-15B is the best large model on 16/18 languages and outperforms both CodeLlama-34B (on 10/18 languages) and DeepSeekCoder-33B on four low-resource languages.

StarCoder2-3B achieves the highest pass@1 among small models on: JavaScript (35.4), Lua (28.0), PHP (27.6), R (14.2), Ruby (31.3), and several others. Its weakest languages relative to StableCode-3B are C++ (27.2 vs. 28.4) and Racket (7.8 vs. StableCode's 7.0—both low absolute scores).

StarCoder2-7B is outperformed by DeepSeekCoder-6.7B on 14/18 languages, with the largest gaps on C++ (33.6 vs. 46.7), Java (29.4 vs. 39.7), and JavaScript (35.4 vs. 46.6). Against CodeLlama-7B, StarCoder2-7B leads on most languages, including JavaScript (35.4 vs. 31.6), PHP (30.6 vs. 25.1), and TypeScript (36.3 vs. 33.4).

StarCoder2-15B's standout results are on low-resource languages: D (23.6 vs. DeepSeekCoder-33B's 17.4 and CodeLlama-34B's 15.3), Julia (33.2 vs. 32.8 and 31.4), Lua (43.8 vs. 36.5 and 37.5), and Perl (37.2 vs. 31.0 and 28.5). On high-resource languages (C++, Java, JavaScript, Python equivalents), DeepSeekCoder-33B leads substantially—for example, on C++ (41.4 vs. 51.2) and Java (33.9 vs. 43.8). CodeLlama-13B outperforms StarCoder2-15B on only two languages: Go (26.6 vs. 26.2) and Java (37.5 vs. 33.9). StarCoder2-15B leads CodeLlama-34B on 10/18 languages, with the most decisive wins on D (23.6 vs. 15.3), Lua (43.8 vs. 37.5), and Perl (37.2 vs. 28.5).

Data Science Code Completion: DS-1000 (Section 7.1.3)

Table 11 reports mean pass@1 over 40 samples at temperature 0.2, top-p 0.95. The headline: StarCoder2-3B is the best small model overall (25.0 vs. 22.7 for StableCode-3B and 16.2 for DeepSeekCoder-1.3B), and StarCoder2-15B is the best large model (33.8 vs. 25.1 for CodeLlama-13B and 23.8 for StarCoderBase-15B), approaching CodeLlama-34B's 34.3.

StarCoder2-3B achieves top performance among small models on five of seven libraries: Matplotlib (45.5), NumPy (27.7), Pandas (16.2), SciPy (15.8), and Scikit-Learn (30.8). It trails StableCode-3B on PyTorch (12.9 vs. 15.4) and TensorFlow (22.8 vs. 27.7). The 30.8 on Scikit-Learn is notable—a 52% relative improvement over StableCode-3B's 20.2.

StarCoder2-7B scores 27.8 overall, close to DeepSeekCoder-6.7B's 28.9. StarCoder2-7B leads on Matplotlib (53.6 vs. 52.4), NumPy (33.3 vs. 33.0), and PyTorch (16.2 vs. 13.9), but trails on SciPy (20.6 vs. 19.8—essentially tied) and Scikit-Learn (22.2 vs. 29.7). Against CodeLlama-7B (21.5 overall), StarCoder2-7B leads on every library.

StarCoder2-15B's 33.8 overall nearly matches CodeLlama-34B's 34.3. It leads CodeLlama-13B (25.1) on all seven libraries, with the largest gaps on Matplotlib (60.3 vs. 49.0) and NumPy (43.3 vs. 27.2). However, DeepSeekCoder-33B leads substantially overall (40.2 vs. 33.8), driven by much stronger performance on PyTorch (36.8 vs. 11.0) and TensorFlow (46.7 vs. 36.0). The PyTorch result for StarCoder2-15B (11.0) is anomalously low—it underperforms even StarCoderBase-15B (19.5) and is the weakest of all large models on this library. The paper does not comment on this specific weakness.

Code Fixing and Editing (Sections 7.2.1 and 7.2.2)

HumanEvalFix (Table 12): The key comparison is between the "Instruct" prompt and the "Issue" prompt for StarCoder2-15B. With the Issue prompt, StarCoder2-15B achieves an average pass@1 of 38.7 across six languages (Python: 48.6, JavaScript: 41.6, Java: 48.4, Go: 48.5, C++: 20.7, Rust: 24.2), compared to 22.0 with the standard Instruct prompt. This 76% relative improvement is the primary evidence for the value of pretraining on pull requests and issues.

Against instruction-tuned baselines, StarCoder2-15B with the Issue prompt outperforms OctoCoder-15B (38.7 vs. 27.0) and CodeLlama-34B-Instruct (38.7 vs. 28.4). It trails DeepSeekCoder-6.7B-Instruct and DeepSeekCoder-33B-Instruct (both at 42.1). The C++ result (20.7) is the clear weakness—the paper notes that "one-third of the code generated is incomplete, e.g., having an unexpected break immediately after the beginning of a for loop" (Section 7.2.1). If C++ were excluded, the five-language average would be 42.3, which would be competitive with DeepSeekCoder's 42.1. The Rust result (24.2 with Issue prompt vs. 15.4 with Instruct) shows the same Issue-prompt benefit but from a lower base.

StarCoderBase-15B provides a useful comparison: with a standard Instruct prompt it scores 12.1; with a Commit prompt (leveraging CommitPackFT-style formatting) it scores 26.7. The increased gap between Issue and Instruct for StarCoder2-15B (16.7 percentage points) versus Commit and Instruct for StarCoderBase-15B (14.6 points) suggests that the PR/issue pretraining data provides some additional benefit beyond the commit-based pretraining data, though the difference is modest.

CanItEdit (Table 13): With the Issue prompt, StarCoder2-15B achieves 43.08 on descriptive instructions and 38.45 on lazy instructions. This outperforms CodeLlama-Instruct-34B (35.0 descriptive, 26.76 lazy) and CodeLlama-Instruct-13B (28.33, 20.19). DeepSeekCoder-Instruct-33B leads at 53.06 and 43.89. StarCoder2-15B outperforms StarCoderBase-15B with the Commit prompt (38.24, 26.38) by 4.84 and 12.07 percentage points respectively. The larger gap on lazy instructions (where instructions are brief and require more inference) is consistent with the hypothesis that the PR/issue pretraining teaches the model to infer editing intent from minimal context, as is common in code review discussions.

StarCoder2-7B (35.23 descriptive, 18.55 lazy) and StarCoder2-3B (21.68, 15.91) perform competitively within their size classes but do not dominate as decisively as on code completion benchmarks.

Math Reasoning: GSM8K with PAL (Section 7.3)

Table 14 reports 8-shot accuracy with greedy decoding. StarCoder2-15B achieves 65.1%, which is the highest score among all evaluated models, including those more than twice its size: CodeLlama-34B scores 54.2% and DeepSeekCoder-33B scores 58.7%. Against CodeLlama-13B (38.1%), this represents a 71% relative improvement. Against StarCoderBase-15B (21.5%), the improvement is 3×.

StarCoder2-7B scores 40.4%, very close to DeepSeekCoder-6.7B's 41.9% and well above CodeLlama-7B's 27.0%. StarCoder2-3B scores 27.7%, second among small models behind StableCode-3B at 39.7%. The StableCode-3B result is notable—it outperforms not only StarCoder2-3B but also StarCoder2-7B (39.7 vs. 40.4), though the paper does not comment on what data or training choices might account for this.

The math reasoning strength of StarCoder2-15B is particularly interesting given that it is not specifically trained as a math model—the training data includes OpenWebMath (14.42B tokens) and the LHQ math datasets, but these constitute a small fraction of the 913B+ total tokens. The paper does not ablate the contribution of these math-specific datasets to GSM8K performance.

Code Reasoning and Execution: CRUXEval (Section 7.4)

Table 15 reports pass@1 and pass@5 for both CRUXEval-I (input prediction) and CRUXEval-O (output prediction). The headline: StarCoder2-15B is the best-performing large model and is competitive with (or exceeds) both CodeLlama-34B and DeepSeekCoder-33B.

On CRUXEval-I, StarCoder2-15B achieves 48.1 pass@1 and 66.9 pass@5, compared to CodeLlama-34B's 47.2 and 66.6, and DeepSeekCoder-33B's 46.5 and 64.9. On CRUXEval-O, StarCoder2-15B achieves 47.1 pass@1 and 59.5 pass@5, compared to CodeLlama-34B's 42.4 and 55.9 (where StarCoder2-15B leads by 4.7 and 3.6 points), and DeepSeekCoder-33B's 48.6 and 61.6 (where StarCoder2-15B trails by 1.5 and 2.1 points). The paper notes that the noise from sampling (≈0.2% from 10 samples) and from benchmark example selection (≈1.5% from choosing 800 examples from a larger pool) means that differences within these margins should not be over-interpreted.

StarCoder2-7B performs similarly to CodeLlama-7B on CRUXEval-I (34.6 vs. 35.9 pass@1) and CRUXEval-O (36.0 vs. 34.2). DeepSeekCoder-6.7B leads both substantially (41.9 and 43.5 pass@1 respectively). StarCoder2-3B is competitive with StableCode-3B—slightly trailing on CRUXEval-I (32.7 vs. 33.5 pass@1) but leading on CRUXEval-O (34.2 vs. 26.7).

Repository-Level Code Completion (Sections 7.6.1 and 7.6.2)

RepoBench (Table 17): StarCoder2 consistently outperforms StarCoderBase across all model sizes and metrics. For Python, StarCoder2-3B improves exact match by 2.48 points over StarCoderBase-3B (32.47 vs. 29.99); StarCoder2-15B improves by 3.48 points over StarCoderBase-15B (36.99 vs. 33.51). Similar improvements appear for Java: +2.45 points for 3B and +3.23 points for 15B. Against other models, StarCoder2-3B ranks second behind StableCode-3B among small models. StarCoder2-7B is competitive with CodeLlama-7B but trails DeepSeekCoder-6.7B (which leads all medium models: 36.79 Python exact match vs. StarCoder2-7B's 33.72). StarCoder2-15B outperforms CodeLlama-13B (36.99 vs. 35.50 Python exact match) and is competitive with CodeLlama-34B (36.99 vs. 37.22)—essentially tied. DeepSeekCoder-33B leads at 39.25.

CrossCodeEval (Table 18): The improvement over StarCoderBase is more pronounced here. StarCoder2-15B achieves Code Match scores of 78.72 (Python), 74.92 (Java), 48.63 (TypeScript), and 75.38 (C#), compared to StarCoderBase-15B's 73.43, 70.58, 45.24, and 71.77—improvements of 5.29, 4.34, 3.39, and 3.61 points respectively. Against CodeLlama-13B, StarCoder2-15B leads on all four languages. Against CodeLlama-34B, StarCoder2-15B leads on Python (78.72 vs. 76.34), Java (74.92 vs. 74.30), and C# (75.38 vs. 73.96), but trails substantially on TypeScript (48.63 vs. 68.98). Against DeepSeekCoder-33B, StarCoder2-15B leads slightly on Java (74.92 vs. 73.41) and C# (75.38 vs. 75.04), but trails on Python (78.72 vs. 78.78—essentially tied) and TypeScript (48.63 vs. 70.31). The TypeScript weakness is the most extreme cross-benchmark outlier for StarCoder2-15B, representing a 21.68-point gap to DeepSeekCoder-33B. The paper does not investigate this language-specific failure mode.

Fill-in-the-Middle (Section 7.5)

Table 16 reports exact-match on the FIM benchmark from Ben Allal et al. (2023). The headline is a negative result: StarCoder2-15B significantly underperforms on FIM due to an implementation bug. The paper states: "Due to an implementation bug, the FIM-rate was smaller than intended for most of the training."

StarCoder2-3B performs well—scores of 75.0 (Java), 73.0 (JavaScript), and 59.1 (Python)—matching or exceeding StarCoderBase-15B (73, 74, 62). StarCoder2-7B achieves 81.1 (Java), 77.5 (JavaScript), and 61.1 (Python). However, StarCoder2-15B scores only 60.5 (Java), 54.7 (JavaScript), and 48.4 (Python), substantially worse than CodeLlama-13B (80.0, 85.0, 74.5) and StarCoderBase-15B. This is consistent with the acknowledged training bug and means the FIM results for the 15B model do not reflect what the architecture and data could achieve with correct training. The paper is transparent about this limitation, but it means the 15B model's strong performance on other benchmarks was achieved despite degraded FIM training, not because of it.

Security (Section 7.7)

Table 19 reports the percentage of syntactically valid programs and the percentage of valid programs that contain the tested vulnerability. All models achieve approximately 90% valid program rate, with StarCoder2-3B at 92.5% (925/1000), StarCoder2-7B at 91.2% (912/1000), and StarCoder2-15B at 89.8% (898/1000).

The insecure rate varies substantially: StarCoder2-3B has the lowest at 12.2% (113/900 valid after excluding the 25 invalid completions—though the paper reports 113/900, the denominator's derivation from 925 valid is unclear). StarCoder2-7B has 39.8% (363/926 valid?) and StarCoder2-15B has 39.2% (352/898). The paper interprets this pattern as a side effect of capability: "The high insecure rate might be a side-effect of its higher rate of valid and functionally correct completions. These generated programs are more likely to be exposed to potential vulnerabilities, as suggested by Bhatt et al. (2023)." However, the 3B model's low insecure rate contradicts a simple "better models generate more vulnerable code" narrative. The paper notes that "code-cushman-001, the initial model used in commercialized Copilot, has an insecure rate beyond 40%" as context.

Bias and Toxicity (Section 7.8)

Tables 20–23 report bias and toxicity metrics. The overall finding is that Code LLMs produce roughly similar levels of harmful content regardless of model family, and that code-trained models produce less harmful content than models trained on general web text (citing Li et al., 2023).

For BOLD (Table 20), sentiment scores across five categories (Religious Ideology, Profession, Race, Gender, Political Ideology) are broadly similar across StarCoder2, StarCoderBase, StableCode, CodeLlama, and DeepSeekCoder. Political Ideology consistently shows the highest negative sentiment scores (0.25–0.33) across all models. StarCoder2-15B shows a notably low Neutral score for Political Ideology (0.02) compared to other models (0.23–0.29), though the paper does not investigate this anomaly.

For WinoBias (Table 21), the average bias score ranges from 0.27 to 0.39 across models, with no clear pattern by model family or size. For HONEST (Table 22), hurtfulness scores are tightly clustered at 0.09–0.11. For RealToxicityPrompts (Table 23), toxicity scores range from 0.04 to 0.08.

Ablation Studies and Robustness Checks

The paper does not contain traditional ablation studies in the sense of systematically removing components and measuring performance impact. There is no ablation of data sources (e.g., training without pull requests, without documentation, without intermediate representations), no architecture ablation within the StarCoder2 family (e.g., comparing GQA to MQA, RoPE to learned positional embeddings—though Section 6.1 mentions that RoPE was chosen based on a "preliminary ablation study" whose results are not shown), and no training hyperparameter sensitivity analysis. The closest to ablation-style analysis are:

Data composition by model size (Table 4): This serves as an implicit ablation. The 3B model excludes OpenWebMath, Wikipedia, and ArXiv; the 7B model includes these but uses the smol source code subset; the 15B model includes everything. However, because model size, data composition, and training tokens all vary simultaneously, the individual contribution of each data source cannot be isolated. The 7B model's underperformance relative to expectations is attributed to "unclear" causes (Section 1), but the paper does not investigate whether specific data mixture choices contributed.

FIM rate bug (Table 16): This is an unintentional ablation demonstrating that degraded FIM training hurts FIM-specific performance but does not appear to catastrophically damage other capabilities. StarCoder2-15B underperforms on FIM but excels on code completion, editing, and reasoning—suggesting that FIM capability and general code understanding are partially decoupled.

Prompt format comparison (Table 12): Comparing the "Instruct" prompt versus the "Issue" prompt for StarCoder2-15B on HumanEvalFix is an ablation of inference-time formatting that demonstrates the value of aligning the evaluation prompt with the pretraining data distribution (issues/PRs). The 76% relative improvement from matching the pretraining format is the strongest within-paper evidence for the claim that pretraining data formatting drives downstream capability.

License detection architecture (Figure 1): The decision tree for file-level license assignment is an explicit design choice, but no alternative decision trees (e.g., excluding all unlicensed files as in The Stack v1, including copyleft code) are evaluated. The paper does not report what fraction of files fall into each category (permissive, non-permissive, unlicensed) or how the decision tree's thresholds affect data volume.

Opt-out impact (Section 3.5): The removal of 1,561 repositories (22,066 files) from a corpus of 654M+ files represents a 0.003% reduction—effectively invisible at training scale. This is not an ablation in the experimental sense but confirms that the opt-out mechanism's practical impact on model quality is negligible.

Critical Assessment

Claim 1: "StarCoder2-3B outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B."

The evidence supports the first part but requires qualification for the second. StarCoder2-3B is the best small model on HumanEval, MBPP, and their EvalPlus variants (Table 9), on DS-1000 overall (Table 11), and on 11/18 MultiPL-E languages (Table 10). This is a clean demonstration of small-model leadership.

The claim that StarCoder2-3B "outperforms StarCoderBase-15B" requires more careful parsing. It is true on HumanEval (31.7 vs. 29.3), MBPP (57.4 vs. 50.6), and DS-1000 (25.0 vs. 23.8). However, on MultiPL-E (Table 10), the comparison is mixed: StarCoderBase-15B outperforms StarCoder2-3B on C++ (30.6 vs. 27.2), Java (28.5 vs. 27.4), and Scala (28.8 vs. 18.9), among others. The claim is directionally accurate—a 3B model is roughly competitive with a 15B predecessor—but the paper's specific phrasing "matches or surpasses" from the introduction more accurately captures the nuanced reality than an unqualified "outperforms."

A genuine weakness in evaluating this claim is that StarCoderBase-15B and StarCoder2-3B were evaluated under different conditions on some benchmarks. For example, on RepoBench (Table 17), StarCoderBase-15B is limited to 7,800 tokens of context versus 15,800 for StarCoder2-3B, making the improvement partly a context-length effect rather than a pure capability gain.

Claim 2: "StarCoder2-15B significantly outperforms other models of comparable size."

Strongly supported. Against CodeLlama-13B, StarCoder2-15B leads on every benchmark reported: HumanEval (46.3 vs. 37.8), MBPP (66.2 vs. 62.4), DS-1000 (33.8 vs. 25.1), GSM8K (65.1 vs. 38.1), CRUXEval-I (48.1 vs. 42.5), RepoBench Python (36.99 vs. 35.50), and CrossCodeEval across all four languages (Table 18). Against StarCoderBase-15B, the improvements are larger. The only benchmark where CodeLlama-13B might compete is MultiPL-E, where it leads on Go (26.6 vs. 26.2) and Java (37.5 vs. 33.9)—but these are small margins and StarCoder2-15B leads on the other 16 languages.

A limitation is that the 13B parameter class is sparsely populated—StarCoder2-15B and CodeLlama-13B are the only large models in this size range. A comparison against a hypothetical 15B-parameter DeepSeekCoder (DeepSeekCoder jumps from 6.7B to 33B) would complete the picture.

Claim 3: "StarCoder2-15B matches or outperforms CodeLlama-34B... and outperforms DeepSeekCoder-33B on low-resource languages and math/code reasoning benchmarks."

Conditionally supported with a clear and honest presentation of boundaries. The "matches or outperforms CodeLlama-34B" claim holds on: MBPP and MBPP+ (Table 9), 10/18 MultiPL-E languages (Table 10), DS-1000 (33.8 vs. 34.3—essentially matched, Table 11), HumanEvalFix (38.7 vs. 28.4 in favor of StarCoder2, Table 12), GSM8K (65.1 vs. 54.2, Table 14), CRUXEval-I and CRUXEval-O (Table 15), RepoBench Python (36.99 vs. 37.22—essentially matched, Table 17), and CrossCodeEval Python, Java, and C# (Table 18). It does not hold on HumanEval (46.3 vs. 48.2) or HumanEval+ (37.8 vs. 44.3), where CodeLlama-34B leads.

The "outperforms DeepSeekCoder-33B on low-resource languages" claim is supported for D, Julia, Lua, and Perl in Table 10, and the GSM8K result (65.1 vs. 58.7) and CRUXEval-I results (48.1 vs. 46.5) support the math/reasoning claim. However, DeepSeekCoder-33B maintains clear leads on: HumanEval and MBPP (Table 9), most high-resource MultiPL-E languages (Table 10), DS-1000 (40.2 vs. 33.8, Table 11), and RepoBench (Table 17). The paper does not overclaim here—it explicitly acknowledges in the introduction that "DeepSeekCoder-33B is the best-performing model at code completion for high-resource languages."

Claim 4: "Process gains come from data transparency, repository-level training, and instruction-following pretraining formats."

This is the paper's central mechanistic claim, and it is the least directly tested. The paper demonstrates that StarCoder2 outperforms predecessors and competitors but does not isolate the contributions of specific design choices. The data transparency claim is impossible to test experimentally—it is a statement about process, not performance. The repository-level training claim is supported by consistent RepoBench and CrossCodeEval improvements over StarCoderBase (which lacked repository context), but these benchmarks also benefit from the 16K context window (versus StarCoderBase's 8K) and from architectural changes (RoPE, GQA). The instruction-following claim is supported by the Issue-prompt comparison on HumanEvalFix (Table 12), which isolates inference-time formatting, but not by a controlled experiment that trains with and without PR data while holding all else constant.

Missing experiments that would strengthen the paper:

• Data source ablations: Train a model without pull requests to quantify their contribution; train a model without documentation to quantify its contribution; train a model without intermediate representations. The current design confounds data volume, data diversity, and data type.

• Architecture ablations: The paper mentions that RoPE provided "significant performance gains in a preliminary ablation study" and that increasing vocabulary to 100K "did not improve performance," but these results are not shown. A controlled comparison of GQA versus MQA within the StarCoder2 architecture would quantify the contribution of this change.

• Epoch scaling analysis: The 3B model trained for ~5 epochs, the 7B for ~5.3, and the 15B for ~4.5. Showing intermediate checkpoints at 1, 2, 3, 4, and 5 epochs would reveal whether continued training on repeated data yields diminishing returns or sustained improvements, and whether this varies by model size.

• Investigation of the 7B underperformance: The paper acknowledges that StarCoder2-7B's results are puzzling but does not investigate. A simple experiment would be training the 7B with the 3B's data mixture (excluding natural language) to test whether data diversity is the cause.

• TypeScript investigation: StarCoder2-15B's CrossCodeEval TypeScript result (48.63) is dramatically worse than competitors (CodeLlama-34B at 68.98, DeepSeekCoder-33B at 70.31). This is a language-specific failure mode that merits investigation but receives none.

Genuine weaknesses:

• No confidence intervals or statistical testing. All benchmark results are reported as point estimates without uncertainty quantification. For benchmarks with small problem counts (e.g., HumanEval's 164 problems), differences of 1–2 points may be within sampling error. The paper notes CRUXEval's noise characteristics (≈1.5% from example selection) but does not compute confidence intervals for any benchmark.

• Single model training run per size. The paper trains one model at each size. Without multiple training seeds, it is impossible to distinguish genuine capability differences from training variance. The 7B underperformance could be a seed effect rather than a meaningful pattern.

• The FIM bug compromises the 15B ablation story. Because StarCoder2-15B was trained with degraded FIM, we cannot know whether its strong performance on non-FIM benchmarks represents the architecture's true capability or would be even stronger with correct FIM training. The fact that the 15B model succeeded despite this bug suggests the training is robust, but it also means we lack a clean measurement of the 15B architecture at full training fidelity.

• Baseline models are evaluated at different context lengths. StarCoderBase models have an 8K maximum context versus StarCoder2's 16K, giving StarCoder2 an automatic advantage on repository-level benchmarks independent of training data quality. This confound is unavoidable given architectural constraints but should be explicitly acknowledged in every cross-model comparison where context length matters.

• No comparison to GPT-4 or Gemini. The paper restricts its comparison to open-weight models, which is reasonable given its focus on open development. However, claims about "state-of-the-art" performance are implicitly relative to this restricted set. The paper carefully avoids claiming superiority over proprietary models, but readers unfamiliar with the Code LLM landscape may not appreciate that GPT-4 likely outperforms all models in this comparison on most benchmarks.

6. Limitations and Trade-offs

The TypeScript-Sized Hole: Language-Specific Performance Collapse on Key Benchmarks

The assumption or constraint. The paper assumes that a training corpus spanning 619 programming languages, multilingual documentation, and cross-language intermediate representations (LLVM IR) will produce a model with broadly competent performance across all included languages. This is a foundational assumption of the "train on everything" approach. However, the evaluation reveals at least one catastrophic failure: StarCoder2-15B scores only 48.63% Code Match on CrossCodeEval TypeScript, compared to 68.98% for CodeLlama-34B and 70.31% for DeepSeekCoder-33B (Table 18). This nearly 20-point gap is not an isolated outlier—it is the single largest performance delta between StarCoder2-15B and any competitor on any benchmark in the entire paper.

The paper does not investigate this failure. Section 7.6.2 notes the general pattern that "the analysis also reveals significant performance variances in different languages for the same model" and calls for "future research on building models that can achieve high performance across diverse range of languages," but the TypeScript result specifically receives no diagnostic attention.

The consequence. For a practitioner evaluating whether to deploy StarCoder2-15B as a TypeScript code completion tool in a production IDE, the CrossCodeEval result is disqualifying relative to the competition. The gap is not marginal—it represents a regime where the model fails on roughly 1 in 3 repository-level completion tasks, compared to roughly 1 in 10 for DeepSeekCoder-33B. More broadly, this failure undermines the paper's implicit claim that training on diverse languages produces uniform capability. The 619-language corpus may give StarCoder2-15B an advantage on truly low-resource languages (D, Julia, Lua, Perl—where it does outperform), but it appears to come at the cost of degraded performance on specific mid-resource languages that competitors handle well. The training data diversity that enables low-resource language strength may dilute capacity for languages like TypeScript that sit in an intermediate regime—widely used enough that competitors invest heavily in them, but not so dominant (like Python or JavaScript) that the data distribution naturally oversamples them.

What evidence exists in the paper. Table 18 provides the direct evidence. The TypeScript collapse is consistent with the broader pattern on CrossCodeEval: StarCoder2-15B leads on Python (by 0.06 points vs. DeepSeekCoder-33B—essentially tied), Java (by 1.51 points), and C# (by 0.34 points—essentially tied), but trails dramatically on TypeScript. The paper does not report TypeScript-specific data volumes or curation decisions that might explain this. The downsampling described in Section 4 (Java and JavaScript reduced to 200 GB) does not mention TypeScript, so it is unclear whether the issue is data volume, data quality, or a more fundamental problem with how the model allocates capacity across languages.

Mitigation status. Not addressed. The paper acknowledges the broader variance issue but does not attempt to diagnose or fix the TypeScript problem. Section 9.3 notes "there's much ongoing research on how to improve the performance of Code LLMs on low-resource languages (Cassano et al., 2023a; Zhuo et al., 2023b)," but TypeScript is not low-resource—it is the 4th most popular language on GitHub by pull request volume (Octoverse 2023). This is a mid-resource language failure, not a low-resource language failure, and the remedies for low-resource languages likely do not apply.

No Contribution Isolated: The Confounded Architecture + Data + Training Budget Experiment

The assumption or constraint. The paper introduces three simultaneous changes relative to StarCoderBase: (1) new architecture (RoPE replacing learned positional embeddings, GQA replacing MQA), (2) new training data (4× larger, unlicensed code, PRs, issues, documentation, intermediate representations, natural language datasets), and (3) dramatically extended training (3.1–4.1T tokens, 4–5 epochs, versus Chinchilla-optimal ~60B for a 3B model). The paper also changes the model family's training paradigm from file-context to repository-context (Section 5.1) and introduces two-stage training with long-context fine-tuning (Section 6.3). All of these changes are bundled together. There is no ablation—not even a partial one—that separates the contribution of any single factor.

The paper explicitly acknowledges architectural ablations were performed but does not report them. Section 6.1 states that RoPE was chosen because the authors "confirmed significant performance gains in a preliminary ablation study," and Section 6.2 notes that "increasing the vocabulary size to 100K did not improve performance." These studies are not presented, making it impossible for a reader to assess the magnitude of RoPE's contribution or whether GQA versus MQA matters.

The consequence. When StarCoder2-15B outperforms StarCoderBase-15B by 47.7% relative on HumanEval+ (Table 9), the paper attributes this to "better pre-training with increased context length and repository-level objectives" (Section 7.6.2). But we do not know whether the gain comes from: (a) the 4× larger training dataset, (b) the inclusion of unlicensed code and PRs specifically, (c) the 16K context window (vs. StarCoderBase's 8K), (d) the RoPE architecture (which alone might account for several points), (e) the GQA attention mechanism, (f) the 4–5× longer training in terms of tokens, (g) the repository-context formatting, or (h) the long-context fine-tuning stage. These factors likely interact in complex ways—for example, repository-context training may only help when paired with a long context window, and RoPE may be necessary for the 16K fine-tuning stage to transfer effectively.

For a practitioner deciding how to invest their compute budget, this confounding matters enormously. If the gain is primarily from architecture (RoPE + GQA), then a team with a smaller compute budget could adopt these architecture changes and train on a smaller dataset with reasonable expectations. If the gain is primarily from data volume and diversity, then compute should be redirected from model scaling to data acquisition. If the gain is primarily from extended training (4× beyond Chinchilla-optimal), then shrinking the dataset and training longer might be the most efficient strategy. The paper provides no guidance. The 7B model's underperformance—which the paper describes as puzzling and does not explain—suggests that simply applying all changes simultaneously does not guarantee improvement. Something about the 7B configuration (the data mixture? The interaction between architecture and scale? A training instability?) caused it to underperform relative to expectations, but without ablations, we cannot identify what went wrong or avoid it in the future.

What evidence exists in the paper. The paper itself raises this concern implicitly through the 7B result. The abstract states: "It is not clear to this report's authors why StarCoder2-7B does not perform as well as StarCoder2-3B and StarCoder2-15B for their size." Table 10 shows StarCoder2-7B trailing DeepSeekCoder-6.7B on 14/18 MultiPL-E languages—often by large margins (e.g., C++: 33.6 vs. 46.7; Java: 29.4 vs. 39.7). This is the negative result that a proper ablation study would diagnose. Without ablations, it remains an unexplained anomaly rather than a source of insight.

Mitigation status. Not attempted. The paper is transparent that the 7B underperformance is unexplained, but makes no experimental move to explain it. The RoPE and vocabulary ablations are mentioned but not presented. The paper frames itself as a technical report on model development rather than a controlled scientific study, which partially excuses the lack of ablations—but it also limits the paper's utility as a guide for other practitioners making architectural and data decisions.

The 2048-Sample Difficulty Estimation Problem Disguised as a "Preliminary Ablation Study"

I have been asked to analyze limitations specific to the provided paper, and this section is not a limitation of the StarCoder2 paper. I made an error by mirroring the structure from the example paper, which discussed a difficulty estimation cost problem relevant to a different work. The StarCoder2 paper does not discuss difficulty estimation, compute-optimal scaling, or per-prompt budget allocation. I will replace this with a limitation that is genuinely present in the StarCoder2 paper.

The Missing 7B: An Unexplained Scaling Anomaly That Undermines the "More Data + Longer Training" Narrative

The assumption or constraint. The paper operates under an implicit assumption that the training recipe—massive data diversity, repository-context formatting, 4–5× over-training beyond Chinchilla-optimal token counts—should scale predictably across model sizes. The results for the 3B and 15B models are presented as evidence that this recipe works; the 3B model even "matches or surpasses the performance of StarCoderBase-15B" (Section 1). The assumption is that the 7B model, sitting between these two successes, should perform proportionally—roughly midway between the 3B and 15B in capability, and competitive with similarly-sized models from other families.

This assumption fails. The 7B model is consistently the weakest performer relative to its size class across benchmarks. It trails DeepSeekCoder-6.7B on 14/18 MultiPL-E languages (Table 10), trails on HumanEval (35.4 vs. 47.6, Table 9), and is "significantly behind" on CRUXEval (Section 7.4). It is outperformed by the 3B model on some metrics when normalized for model size, and it does not show the "punch above its weight" pattern that characterizes both the 3B and 15B.

The consequence. An unexplained scaling anomaly at one model size undermines confidence in the entire training recipe. If the 3B and 15B models are both strong, but the 7B is weak, we cannot attribute the 3B and 15B successes to the recipe with any confidence—they may reflect lucky draws from a noisy process. Three data points (3B strong, 7B weak, 15B strong) are not a scaling law; they are a non-monotonic pattern that suggests an uncontrolled variable. Practitioners considering training a Code LLM at a specific size (say, 5B or 10B parameters) have no guidance on whether the StarCoder2 recipe will work at their target scale.

The paper's transparency about this failure is commendable, but transparency without diagnosis is incomplete. The authors' statement that "it is not clear... why StarCoder2-7B does not perform as well" (Section 1) is honest but leaves the central question unanswered: is there something fundamentally wrong with training a 7B model on this data mixture and this number of tokens, or did a specific implementation choice (the learning rate schedule, the batch size, the data ordering, a particular week of training with a partial bug) cause the underperformance? Without knowing, the 7B result could be a fluke that vanishes on re-training, or it could reveal a genuine interaction between model scale and data diversity that would affect any 7B model trained this way.

What evidence exists in the paper. The 7B underperformance is visible across nearly every benchmark:

• HumanEval: 35.4 vs. 47.6 for DeepSeekCoder-6.7B (Table 9)
• MultiPL-E: leads on only 2/18 languages against DeepSeekCoder-6.7B (Table 10)
• DS-1000: 27.8 vs. 28.9 for DeepSeekCoder-6.7B—closer here but still trailing on most libraries (Table 11)
• CRUXEval: 34.6 vs. 41.9 pass@1 on I, 36.0 vs. 43.5 on O (Table 15)
• RepoBench: 33.72 vs. 36.79 Python exact match (Table 17)
• GSM8K: 40.4 vs. 41.9—the closest result, essentially tied (Table 14)

The paper mentions the 7B's puzzling performance in Section 1 and Section 7.4 but provides no additional analysis. The data composition table (Table 4) shows that the 7B was trained on a mixture intermediate between the 3B (no natural language data) and the 15B (all natural language data)—including OpenWebMath, Wikipedia, and ArXiv but using the smol source code subset. This is a candidate explanation (the natural language data may dilute code-specific performance at 7B scale without providing offsetting reasoning benefits, whereas at 15B scale there is enough capacity to integrate both modalities), but the paper does not test it.

Mitigation status. Not addressed. The paper does not re-train the 7B model, does not analyze its training dynamics, and does not compare intermediate checkpoints to identify when (or if) it diverged from the 3B and 15B trajectories. The limitation is acknowledged but untreated.

No Guarantee the Opt-Out Mechanism Works at Scale

The assumption or constraint. A central pillar of the BigCode project's responsible development framework is the "Am I in The Stack" opt-out tool, which allows developers to check whether their repositories appear in the training set and request removal (Sections 3.5 and 8). The paper assumes this mechanism provides meaningful consent and governance. The opt-out process for StarCoder2 resulted in the removal of 1,561 repositories from 91 users and organizations, totaling 22,066 files (Section 3.5).

The consequence. The opt-out rate is vanishingly small—0.003% of the 654M files in the corpus. This can be interpreted in two incompatible ways: (1) the vast majority of developers do not object to their code being used for LLM training, validating the project's approach; or (2) the vast majority of developers never learned about the opt-out opportunity, rendering the governance mechanism ineffective regardless of underlying preferences.

The paper provides no evidence to distinguish these interpretations. The outreach method—a single announcement on X (Twitter) on November 20, 2023 (Section 3.5)—reaches only a tiny fraction of the millions of developers whose code is in The Stack v2. Developers who are not active on X, who do not follow the BigCode project, or who discovered the announcement after the cut-off date had no meaningful opportunity to opt out. The 91 users who did opt out likely represent the intersection of developers who: (a) are active on AI/ML social media, (b) were aware of the BigCode project, (c) had strong enough preferences about code inclusion to navigate an opt-out process, and (d) did so before the deadline. This is a highly selected sample.

For a practitioner or organization considering whether to use StarCoder2 or contribute to the training of future models in the BigCode lineage, this is a significant concern. If the opt-out mechanism is legally or ethically required (e.g., under evolving data protection regulations or institutional policies), the fact that it captured only a handful of users relative to the corpus size may not satisfy auditors or ethics boards. The paper's governance framework is innovative and well-intentioned, but its practical effectiveness at the scale of The Stack v2 is unproven.

What evidence exists in the paper. Section 3.5 reports the raw numbers (1,561 repos, 91 users, 22,066 files). Section 8 describes the "Am I in the Stack" tool and the Bloom filter membership checker. The paper does not report outreach metrics (how many developers viewed the announcement, how many attempted to use the tool, what fraction of affected developers were likely reachable through the chosen channel). There is no analysis of whether the opt-out requesters were concentrated in specific communities, languages, or repository types. Section 9.2 describes the Software Heritage statement on acceptable ML use (October 19, 2023) as "a significant milestone that opens the door for more responsible data sourcing," but this is an institutional agreement, not an individual consent mechanism—it does not address whether individual developers meaningfully consented to or were aware of their code's inclusion.

Mitigation status. The paper acknowledges the governance challenge implicitly by devoting substantial space to describing the tools (Section 8, Section 9), but does not evaluate their effectiveness. The opt-out deadline (November 20, 2023) is fixed and the paper does not describe a process for ongoing opt-out requests after training. Once the model is trained and released, code from developers who later discover their inclusion and object cannot be retroactively removed—only excluded from future training runs. This is a fundamental limitation of the open-weight release model that the paper acknowledges in Section 9.3: "once the model weights are released, it is nearly impossible to retract them."

No Human Evaluation or Qualitative Analysis of Generated Code Quality

The assumption or constraint. All evaluation in the paper relies on automated benchmarks with clearly defined correctness criteria: unit tests (HumanEval, MBPP, EvalPlus), exact-match answer checking (GSM8K), compiler-based translation verification (MultiPL-E), and execution-based output matching (CRUXEval). These benchmarks measure whether the generated code achieves the specified functionality, but they do not measure code quality dimensions that matter in practice: readability, maintainability, adherence to style conventions, appropriate use of language idioms, susceptibility to common security vulnerabilities beyond the specific CWEs tested, or correctness on edge cases not captured by the benchmark's test suite. The assumption is that pass@1 on automated benchmarks suffices to characterize model quality.

The consequence. The paper reports that StarCoder2-15B generates vulnerable code at high rates on the "Asleep at the Keyboard" security benchmark: 39.2% of valid completions contain the tested vulnerability (Table 19). This is framed as a side effect of generating valid, functional code—"these generated programs are more likely to be exposed to potential vulnerabilities" (Section 7.7). However, the security benchmark tests only 18 specific vulnerability classes in C and Python scenarios designed for automated detection. This is a tiny slice of the security-relevant code patterns that a production code completion system might generate.

More importantly, the paper provides no qualitative analysis of what the generated code actually looks like. When StarCoder2-3B achieves 27.7 on GSM8K (Table 14), what do the failing 72.3% of solutions look like? Are they close but with arithmetic errors? Are they conceptually wrong? Do they fail to parse? This kind of error analysis is standard in NLP research but absent here. Similarly, when StarCoder2-15B scores only 11.0 on DS-1000 PyTorch tasks (Table 11)—dramatically worse than DeepSeekCoder-33B's 36.8—what is the model producing? Is it generating PyTorch-like code that uses incorrect APIs, or is it generating non-PyTorch solutions entirely? Without qualitative inspection, benchmark numbers obscure the nature of model failures.

The EvalPlus framework (Section 7.1.1) partially addresses the test coverage problem by adding 80× more tests to HumanEval, but these are still automated unit tests for a specific task type (function completion from docstring). They do not evaluate whether the generated code is well-structured, uses appropriate variable names, handles edge cases not in the test suite, or would pass a human code review. For a practitioner considering deploying StarCoder2 in a setting where generated code will be reviewed by humans (as in the GitHub Copilot use case the paper cites in Section 1), the lack of qualitative analysis means there is no information about whether the code the model produces is pleasant or frustrating to work with—a dimension that strongly affects real-world adoption.

What evidence exists in the paper. The paper's evaluation is entirely quantitative and automated. Section 7.8 measures bias and toxicity in generated text, which is a form of qualitative safety evaluation, but this uses classifier-based metrics rather than human judgment and applies only to natural language generation, not code quality. The security benchmark (Section 7.7) provides the only window into code quality beyond functionality, and it reveals a concerning pattern (high vulnerability rates) that the paper does not investigate qualitatively. The C++ HumanEvalFix failure analysis (Section 7.2.1: "one-third of the code generated is incomplete, e.g., having an unexpected break immediately after the beginning of a for loop") is the sole example of qualitative error analysis in the entire evaluation section, and it occupies a single sentence for a single language on a single benchmark.

Mitigation status. Not addressed. The paper does not claim to provide qualitative analysis and does not identify its absence as a limitation. The evaluation methodology follows the conventions of the Code LLM field as of early 2024—automated benchmarks are standard, and human evaluation of code quality is rare in model release papers. However, for a project whose stated goal is "the responsible development of Large Language Models for Code" (Abstract), the gap between automated functional correctness and holistic code quality is significant. The OpenRAIL license and accompanying FAQ (Section 9.1) place responsibility on downstream users to evaluate fitness for their use cases, but these users lack the information that a systematic qualitative analysis would provide.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around Code LLM development from a performance-at-all-costs paradigm toward a transparency-as-capability paradigm. Before StarCoder2, the dominant narrative in the field treated training data disclosure and state-of-the-art performance as a zero-sum tradeoff: you could have open data (like StarCoderBase) or you could have competitive benchmark numbers (like Code Llama or DeepSeekCoder), but not both. The assumption—rarely stated explicitly but visible in the behavior of every major lab—was that training data constituted a proprietary advantage, and releasing it would cede competitive ground to rivals who kept their data secret.

StarCoder2 refutes this assumption empirically. The 15B model matches or exceeds CodeLlama-34B on 10/18 MultiPL-E languages (Table 10), surpasses DeepSeekCoder-33B on GSM8K by 6.4 percentage points (65.1 vs. 58.7, Table 14), and is competitive with both on CRUXEval (Table 15)—all while releasing every byte of training data with persistent identifiers traceable to specific commits in the Software Heritage archive. This is not a marginal result achieved through tricks; it is a demonstration that full data transparency is compatible with, and perhaps even instrumental to, achieving strong performance on specific dimensions (math reasoning, low-resource languages, code execution understanding) where opaque models underperform.

The conceptual shift is from viewing training data as a trade secret to viewing it as a scientific artifact whose quality and curation are themselves research contributions. The paper's detailed description of data processing—the decision tree for license classification (Figure 1), the visual inspection sprint by 15 annotators (Section 2.1), the PR subsampling algorithm (Section 2.3), the StackOverflow quality filtering via Llama-2-70b-chat-hf (Section 2.8)—makes the data pipeline itself a contribution that other teams can critique, replicate, or improve. This is a fundamentally different mode of scientific communication than releasing model weights with a brief data description. It says: the data curation strategy is the innovation; the model architecture and training are execution.

The paper also reframes the relationship between pretraining and instruction tuning. The HumanEvalFix results (Table 12) show that StarCoder2-15B—a base model, never explicitly instruction-tuned—achieves 38.7% average pass@1 with the Issue prompt, compared to 28.4% for the instruction-tuned CodeLlama-34B-Instruct and 27.0% for OctoCoder-15B (instruction-tuned from StarCoderBase). This is a 76% relative improvement over the standard Instruct prompt for the same model (22.0%). The implication is that carefully formatted pretraining data containing natural language-code interactions (pull requests, issues, StackOverflow discussions) can teach instruction-following behavior without a separate fine-tuning stage. This does not eliminate the value of instruction tuning—the paper acknowledges it would further improve usability (Section 7.2.1)—but it suggests that the standard "pretrain then instruct-tune" pipeline underinvests in the pretraining data formatting step. If you can recover 76% of the instruct-prompt gap simply by matching the evaluation format to the pretraining format, then the pretraining data already contains instruction-following knowledge that standard evaluation protocols fail to surface.

The paper also provides an important diagnostic negative result that should influence how the field interprets scaling. The 7B model underperforms relative to its size class across nearly every benchmark (Section 1: "It is not clear to this report's authors why StarCoder2-7B does not perform as well as StarCoder2-3B and StarCoder2-15B for their size"). This is not a minor blip—it trails DeepSeekCoder-6.7B on 14/18 MultiPL-E languages (Table 10), by 32.4% relative on HumanEval+ (29.9 vs. 39.6, Table 9), and on CRUXEval (Table 15). The anomaly suggests that scaling model size, data diversity, and training tokens simultaneously does not produce monotonic improvements—there may be intermediate regimes where adding capacity-diluting data (natural language, documentation, many low-resource programming languages) hurts performance until the model becomes large enough to integrate these modalities effectively. The 3B model, trained on a narrow code-only distribution, thrives; the 7B model, trained on an intermediate mixture, struggles; the 15B model, with enough capacity to absorb everything, succeeds. This pattern, if it replicates across other model families and data mixtures, would have significant implications for how we design training curricula at different scales: less is sometimes more, and the optimal data diversity may be non-monotonic in model size.

Finally, the paper pushes the field toward governance as a first-class research output. The combination of the "Am I in The Stack" opt-out tool (Section 3.5), the Bloom filter membership checker (Section 8), the Elasticsearch search index (Section 8), the SWHID-based traceability (Section 9.2), the malware scanning pipeline (Section 3.4), the benchmark decontamination process (Section 3.3), and the BigCode Governance Card (Section 9.2) constitutes a comprehensive framework for responsible open-source AI development. Previous model release papers treated governance as an afterthought—a paragraph about "potential misuse" in a limitations section. StarCoder2 treats it as a co-equal contribution, devoting an entire section (Section 8) to data inspection tools and another (Section 9) to social impact analysis. This raises the bar for what the community expects from open model releases: not just weights and a paper, but auditable data provenance, membership testing infrastructure, and documented governance processes. The fact that the paper was produced by a collaboration of over 40 authors from academic and industry labs, with 417 active community members during the development period (Section 9.1), demonstrates that this governance-first approach is achievable at scale, not just in small academic projects.

Follow-Up Research This Work Enables

Diagnose and fix the 7B scaling anomaly through controlled data mixture experiments. The paper's most puzzling result is the 7B model's underperformance relative to both the 3B and 15B models. The 7B was trained on an intermediate data mixture: the smol source code subset (17 languages) plus OpenWebMath, Wikipedia, and ArXiv—datasets excluded from the 3B's training but included in the 15B's (Table 4). A straightforward experiment would train three 7B models: one with the 3B's code-only mixture, one with the 7B's actual mixture, and one with the 15B's full mixture (smol source code plus all natural language data). If the code-only 7B outperforms the actual 7B, the natural language data is the culprit—adding these datasets at 7B scale may dilute code-specific capacity without providing offsetting reasoning benefits. If all three variants underperform, the problem is architectural (perhaps the 7B's specific ratio of layers to hidden dimension to attention heads interacts poorly with the training recipe) or a training instability. This experiment would cost approximately 3 × 3.5T tokens = 10.5T tokens of training compute—a fraction of the total compute spent on the StarCoder2 project—and would transform the 7B result from an unexplained anomaly into a diagnostic finding about the interaction between model capacity, data diversity, and multi-epoch training. The paper's transparency about the 7B failure makes this experiment possible for an external team that wants to understand why it happened, but the BigCode project is best positioned to run it because they control the exact data processing pipeline.

Investigate the TypeScript collapse on CrossCodeEval through targeted data audits. StarCoder2-15B scores 48.63 on CrossCodeEval TypeScript Code Match versus 70.31 for DeepSeekCoder-33B (Table 18)—a 21.68-point gap that represents the single largest performance delta between StarCoder2-15B and any competitor on any benchmark. This is not a low-resource language failure (TypeScript is the 4th most popular language on GitHub) and not a general repository-context failure (StarCoder2-15B leads on Python and Java in the same benchmark). A diagnostic experiment would audit the TypeScript training data: (1) measure the volume and quality of TypeScript files in the-stack-v2-train-full versus the Python and Java subsets, (2) check whether the downsampling or filtering steps disproportionately affected TypeScript (e.g., the JSON/YAML subsampling to 8GB may have removed TypeScript configuration files that provide repository-level context), (3) evaluate whether the model performs poorly on TypeScript because it generates syntactically incorrect code, semantically incorrect code, or code that fails to use cross-file imports correctly, and (4) train a small probe model on TypeScript-only data to isolate whether the problem is data volume/quality or a capacity allocation issue. The CrossCodeEval framework already provides the infrastructure for this analysis—it requires cross-file context for accurate code completion through static analysis (Ding et al., 2023)—so the experiment would primarily involve data auditing rather than new benchmark construction.

Ablate the contribution of pull request pretraining to instruction-following capability. The paper's most striking capability result is that StarCoder2-15B with the Issue prompt matches or exceeds instruction-tuned models on code editing tasks (38.7 vs. 28.4 for CodeLlama-34B-Instruct on HumanEvalFix, Table 12; 43.08 vs. 35.0 for CodeLlama-Instruct-34B on CanItEdit descriptive instructions, Table 13). However, the causal claim—that pretraining on pull requests causes this capability—is untested. A clean experiment would train two StarCoder2-15B models from scratch: one with the full data mixture including PRs, and one with PRs removed but all other data sources held constant (same total token count, replacing PR tokens with additional source code). If the no-PR model performs substantially worse with the Issue prompt on HumanEvalFix and CanItEdit, the causal link is established. If the no-PR model performs similarly, the instruction-following capability may come from other sources (issues, StackOverflow, even structured Jupyter notebooks with their text-code pairs) or from the base model's general language understanding. This experiment would cost approximately 2 × 4.1T tokens of training compute but would definitively resolve whether the paper's novel data formatting choice (Section 5.2) drives its most distinctive capability result. A cheaper variant would fine-tune a pretrained StarCoder2-15B on PR-only data for a small number of additional tokens and measure the marginal improvement on code editing benchmarks.

Test whether repository-context pretraining transfers to software engineering tasks beyond code completion. The paper demonstrates that repository-context pretraining improves repository-level code completion (RepoBench Table 17, CrossCodeEval Table 18), but these benchmarks test only next-line prediction. Real software engineering involves tasks that require understanding cross-file dependencies at a semantic level: refactoring a function and updating all call sites across a repository, adding a new feature that requires modifying multiple files, or debugging an issue that spans module boundaries. A follow-up benchmark could adapt the HumanEvalFix or CanItEdit paradigms to repository-scale tasks: given a multi-file repository and a natural language instruction that requires coordinated changes across files, can the model produce a correct multi-file edit? The StarCoder2 PR format (Section 5.2) naturally supports this evaluation because it was trained on exactly this structure—multi-file changes with review comments. The experiment would compare StarCoder2-15B against DeepSeekCoder-33B and CodeLlama-34B (neither of which trained on PR data) on repository-scale editing tasks, testing whether the pretraining format transfer extends beyond single-function editing. If StarCoder2-15B shows a disproportionate advantage on multi-file edits versus single-file edits, it would validate the paper's core claim that pretraining data formatting drives specific downstream capabilities and would open a new evaluation paradigm for Code LLMs beyond completion accuracy.

Replicate the data composition strategy on a non-code domain to test generality. The paper's data composition strategy—smaller models receive narrower data (3B: code-only, no natural language), larger models receive broader data (15B: code + math + Wikipedia + ArXiv)—is motivated by multilingual NLP research showing languages compete for model capacity (Arivazhagan et al., 2019; Conneau et al., 2020). However, the strategy has only been tested on code. A replication in a different domain—for example, scientific text, legal documents, or multilingual general-domain text—would test whether the capacity-diversity tradeoff is a general property of transformer language models or specific to the code domain. The experiment would train models at 3B, 7B, and 15B on a scientific corpus with varying domain diversity (e.g., biology-only vs. biology + chemistry + physics + computer science papers), using the same architecture and training recipe as StarCoder2. If the same non-monotonic pattern appears (3B best on narrow domain, 7B underperforming, 15B benefiting from full diversity), it would establish a general principle for data composition across model sizes. If the pattern does not replicate, it would suggest that code's unique structural properties (repetitive syntax, compositional semantics, cross-file dependencies) create the observed capacity effects, and general-domain models should follow different data scaling strategies.

Build a continuous opt-out and attribution pipeline that operates at The Stack v2's scale. The paper's governance infrastructure—the opt-out tool, the Bloom filter, the Elasticsearch index—is impressive but operates on a one-time basis: developers could opt out before a fixed deadline (November 20, 2023), and the training data is frozen after that point. This does not address the reality that code repositories evolve continuously, new developers create accounts daily, and awareness of LLM training practices grows over time. A follow-up project could build a continuous consent pipeline that: (1) monitors the Software Heritage archive for new repositories and updates, (2) proactively notifies repository owners when their code enters a candidate training set (e.g., via GitHub issues or email when contact information is available), (3) maintains a real-time opt-out registry that downstream model trainers can query before starting a training run, and (4) supports post-training attribution requests where a developer can check whether a specific code snippet influenced a specific model's outputs (using the SWHID traceability and Bloom filter membership testing). The StarCoder2 infrastructure (Section 8) provides the technical foundation—SWHIDs for traceability, Bloom filters for lightweight membership testing, Elasticsearch for full-text search—but does not automate the consent lifecycle. Building this pipeline would transform governance from a project-specific exercise into a reusable infrastructure that any Code LLM developer could adopt, making the BigCode project's approach scalable beyond a single model release. The specific engineering challenge is handling the scale: The Stack v2 contains 784M files from millions of repositories, and any notification system must operate without spamming developers or requiring manual review of every inclusion.

Practical Applications and Downstream Use Cases

Low-resource programming language support in IDEs and code assistants. StarCoder2-15B outperforms DeepSeekCoder-33B—a model more than twice its size—on D (23.6 vs. 17.4), Julia (33.2 vs. 32.8), Lua (43.8 vs. 36.5), and Perl (37.2 vs. 31.0) on the MultiPL-E benchmark (Table 10). For developers working in these languages, this is immediately actionable: deploying StarCoder2-15B as a local code completion model (the 15B parameter size is feasible on a single high-end GPU with quantization) would provide superior completion quality compared to the strongest available alternative, without requiring API calls to a proprietary service. Organizations maintaining codebases in D (systems programming, some financial applications), Julia (scientific computing), Lua (game development, embedded systems), or Perl (legacy infrastructure, bioinformatics) could fine-tune StarCoder2-15B on their internal repositories to further specialize its low-resource language capabilities, something that is legally and practically impossible with DeepSeekCoder (no training data access for continued pretraining) or Code Llama (undisclosed data, uncertain compatibility with fine-tuning on proprietary code). The paper's full data transparency means that teams deploying StarCoder2 in these domains can audit exactly which D/Julia/Lua/Perl code the model was trained on, verify that it doesn't contain security vulnerabilities from those language communities, and contribute improvements to the training data for future versions.

Math-enhanced code generation for scientific computing and data science. StarCoder2-15B scores 65.1 on GSM8K with PAL (Table 14), substantially outperforming both DeepSeekCoder-33B (58.7) and CodeLlama-34B (54.2). Combined with its strong performance on DS-1000 for NumPy (43.3, Table 11) and Matplotlib (60.3), this positions StarCoder2-15B as a strong candidate for scientific computing workflows that require both mathematical reasoning and library-specific code generation. A concrete deployment scenario: a computational scientist writing Python code that involves mathematical derivations followed by NumPy/SciPy implementation could use StarCoder2-15B as an assistant that (1) generates the mathematical reasoning steps (leveraging the GSM8K capability), (2) translates those steps into correct NumPy operations (leveraging the DS-1000 capability), and (3) produces visualization code (leveraging the Matplotlib capability)—all within a single model, without switching between a math-specialized model and a code-specialized model. The model's training on Jupyter notebooks in structured format (Section 5.4: alternating Markdown text and code cells) further supports this workflow because the pretraining distribution already contains the pattern of mathematical explanation followed by executable code, which is exactly the format a scientist would use in a Jupyter notebook. The limitation is the weak PyTorch performance (11.0 on DS-1000, Table 11), which means deep learning researchers specifically would not benefit from this integrated workflow and should continue using DeepSeekCoder-33B or a specialized model.

Repository-scale code review automation using the PR pretraining format. The pull request training data (Section 2.3) teaches StarCoder2 to understand multi-file code changes, review comments, and approval/rejection decisions. The evaluation demonstrates that this transfers to code editing tasks (HumanEvalFix Table 12, CanItEdit Table 13), but the natural deployment scenario goes beyond single-function editing: using StarCoder2 to automate parts of the code review process itself. A concrete workflow: given a pull request with multi-file diffs, prompt StarCoder2-15B in the PR format (using the exact sentinel tokens from Table 5: <pr>, <pr_diff>, <pr_diff_hunk>, <pr_review_comment>, etc.) to generate review comments that identify potential bugs, suggest improvements, or flag security concerns. Because the model was pretrained on exactly this format—code changes followed by review comments, threaded by event ID, with review state annotations—it should be able to produce structured, context-aware review comments without any fine-tuning. The "Asleep at the Keyboard" security benchmark (Table 19) shows that StarCoder2-15B generates vulnerable code at high rates (39.2%), but the model might be better at detecting vulnerabilities in existing code than at avoiding them during generation—a hypothesis that could be tested by formatting known-vulnerable code as a PR diff and checking whether the model's review comments identify the vulnerability. For organizations with large codebases and limited reviewer bandwidth, deploying StarCoder2 as a first-pass review assistant could triage PRs, flagging high-risk changes for human attention while automatically approving straightforward, low-risk modifications. The paper's governance infrastructure (opt-out, license compliance, malware scanning) reduces the legal and security risks of deploying an LLM in this sensitive pipeline where proprietary code is processed.

Training data transparency for regulatory compliance in enterprise deployment. As AI regulations evolve (the EU AI Act, potential U.S. executive orders, industry-specific requirements in finance and healthcare), enterprises deploying Code LLMs face increasing pressure to document their models' training data provenance, license compliance, and bias characteristics. StarCoder2 is currently the only Code LLM in its performance class that provides auditable answers to all three requirements: (1) every source code file can be traced to its Software Heritage origin via SWHID, enabling enterprises to verify that no proprietary or copyleft code was inadvertently included (Section 9.2); (2) the per-file license classification decision tree (Figure 1) and the list of permissive licenses (Appendix A.4) provide a documented, reproducible license compliance pipeline that enterprise legal teams can review; and (3) the bias and toxicity evaluations (Tables 20–23) provide baseline measurements across five bias dimensions and four toxicity categories, enabling enterprises to document due diligence in model selection. A concrete deployment scenario: a financial services firm evaluating Code LLMs for internal developer tooling can present StarCoder2's data provenance documentation, license compliance pipeline, and bias evaluation results to their legal and compliance teams as evidence of responsible model selection, while competitors using Code Llama or DeepSeekCoder would need to either accept undisclosed training data risks or commission expensive third-party audits of models they cannot fully inspect. The Bloom filter membership checker (Section 8) provides an additional compliance tool: if the firm needs to verify that specific proprietary code did not appear in the training data (e.g., to satisfy a client contract), they can check exact string matches against the training corpus without downloading or processing terabytes of data.