ArXiv: 2407.13739

🎯 Pitch

Open-source Granite code models can now handle 128K tokens of context—a 32x increase—with zero loss on standard coding benchmarks. The secret is a simple RoPE adjustment paired with clever repository-level file packing, proving you don't need expensive training to unlock long-context understanding.

1. Executive Summary

This paper introduces a family of long-context code language models that extend the context window of Granite 3B/8B code models from 2K/4K tokens up to 128K tokens. The approach combines lightweight continual pretraining with a gradually increasing RoPE base frequency and repository-level file packing (organizing files from the same repository into semantically ordered, packed training sequences) followed by instruction tuning on a mix of short and long-context data (using synthetically generated multi-turn instructions bootstrapped from the pretraining corpus). On long-context benchmarks, the 128K models achieve dramatic gains over their short-context predecessors—the 8B base model improves RepoBench-P exact match accuracy by ~36 percentage points at 32K context, and the 8B instruct model reaches 61.6% retrieval accuracy on RepoQA at a 0.5 similarity threshold where the 4K version scores only 0.6%. The paper establishes that these long-context gains come at negligible cost to short-context performance—HumanEvalPack pass@1 degrades by only ~1% on average—demonstrating that context-window extension through lightweight RoPE adjustment and data re-engineering preserves general code generation capability while unlocking repository-scale understanding.

2. Context and Motivation

The Core Problem: Open-Source Code Models Have Impractically Short Context Windows

This paper addresses a specific, practical gap in the open-source code LLM ecosystem: while proprietary models like GPT-4, Gemini, and Claude support very long context windows (often hundreds of thousands of tokens), most open-source code models are restricted to relatively short context lengths — typically 2K to 4K tokens. The Granite code models the authors are extending (Granite-3b-Code-Base-2K and Granite-8b-Code-Base-4K) fall squarely into this category, having been originally pretrained on sequences of only 2K and 4K tokens respectively. This constraint is not unique to Granite — it is a widespread limitation across open-source code models (CodeGemma Team et al., 2024; Rozière et al., 2023).

The consequence is that open-source code models cannot effectively handle repository-level coding tasks. In real-world software development, understanding and generating code often requires reasoning about files distributed across an entire repository — tracking dependencies, understanding API usage patterns across modules, and maintaining consistency with project-wide conventions. A model limited to 2K–4K tokens can at best see one or two source files at a time, making it blind to the broader context that a human developer relies on when navigating a codebase. The authors are explicit about this motivation in Section 1:

"With the emergence and development of repository-level coding tasks (Liu et al., 2024; 2023b) and software development agents (OpenDevin Team, 2024), long context length becomes an important feature for code language models."

The keywords here are repository-level coding tasks and software development agents. The former refers to benchmarks and real-world workflows where the model must comprehend and generate code within a multi-file repository structure — tasks like predicting the next line of code given the full repository context, or finding a specific function implementation buried among hundreds of files. The latter refers to autonomous coding agents (like OpenDevin, SWE-Agent, etc.) that need to read, modify, and reason about entire codebases over extended interactions. Both use cases fundamentally require models that can hold large amounts of code in context simultaneously.

Why This Matters: Practical, Economic, and Strategic Stakes

The importance of this problem extends along several dimensions:

Practical deployment constraints. If open-source code models cannot handle long contexts, then anyone building developer tooling on top of open models is forced into brittle workarounds — chunking repositories into small pieces, maintaining separate context windows for different files, or implementing complex retrieval-augmented generation (RAG) pipelines that approximate the full-context understanding a long-context model would provide natively. These workarounds introduce latency, complexity, and failure modes (e.g., missing cross-file dependencies that a human developer would catch immediately). A model that can simply ingest the entire relevant codebase in one context window eliminates entire categories of engineering complexity.

The open-source vs. proprietary capability gap. Proprietary models like GPT-4 and Claude have established long-context code understanding as a baseline capability that users have come to expect. Organizations that cannot use proprietary APIs — due to data privacy concerns, cost constraints, or vendor lock-in risk — are left with open-source alternatives that fundamentally cannot perform the same class of repository-scale tasks. Closing this gap enables genuinely private, self-hosted code assistants that can operate at the repository level, which is critical for enterprise software development where source code is a sensitive asset.

Economic efficiency of context extension. Perhaps most importantly, the paper demonstrates that extending context length does not require retraining from scratch. The authors' extended models are produced through a lightweight continual pretraining phase that processes only 4 billion additional tokens — roughly 0.1% of the original pretraining data volume. This is a crucial finding for the open-source community because it means context-length extension is economically accessible: organizations with existing pretrained code models can extend their context windows at a tiny fraction of the original training cost, rather than needing to retrain from scratch with long sequences. This massively lowers the barrier to entry for producing long-context code models and suggests that short-context models can be "upgraded" post-hoc as longer-context needs emerge.

Where Prior Approaches Fall Short

The paper's approach is informed by several limitations in the existing landscape of long-context modeling for code:

Sparse and linear attention approximations introduce trade-offs. Some prior long-context approaches rely on sparse attention patterns (attending to only a subset of token pairs) or linear attention mechanisms (reducing the quadratic O(n2)\mathcal{O}(n^2) cost of full attention to O(n)\mathcal{O}(n)) to make long sequences computationally feasible. These methods introduce architectural departure from the base model's full-attention design and often degrade performance on the short-context tasks the model was originally trained for. The Granite models use full attention throughout (Section 2.1), avoiding these architectural compromises. The authors state:

"We continue pretrain the full attention Granite code base models using sequence parallelism... without using any sparse or linear attention."

This design principle — maintain the exact same attention mechanism as the original short-context model — ensures that when the model encounters short sequences, its behavior is as close as possible to the original. The extention is purely through data and positional encoding adjustments rather than architectural surgery.

Scaling RoPE with insufficient data engineering. Prior work (Xiong et al., 2023) established that adjusting the RoPE (Rotary Position Embedding) base frequency can extend the effective context window of transformer models beyond their training length. However, simply cranking up the RoPE theta value without corresponding changes to the training data distribution leads to models that can attend to long contexts in principle but do not learn to use that capacity effectively. The data must contain sufficient long-sequence examples for the model to practice long-range reasoning during training. This is where many prior efforts fall short — they modify the positional encoding but feed the model data distributions that are still dominated by short sequences.

Naive data concatenation loses repository structure. A straightforward approach to creating long-context training data is to simply concatenate documents until a target sequence length is reached. For code, this naive approach breaks critical semantic relationships — files from different repositories get interleaved, import chains are broken, and the model never sees the coherent multi-file structure that characterizes real repositories. The paper identifies this as a key deficiency and develops a repository-level file packing approach that preserves and exploits the dependency structure within repositories. This is described in Section 2.1:

"We develop a new approach that packs files from the same repository together, arranging them to prioritize semantic dependencies. We identify these dependencies by analyzing file imports and create a directed acyclic graph, where each file is a node and edges represent API imports between files."

This transforms the training data from a bag of independent code files into coherent repository snapshots where the model can learn cross-file reasoning patterns. The paper also implements per-language context length upsampling — intentionally oversampling longer documents on a per-programming-language basis — because "long data usually come from particular sources" and without explicit upsampling, longer sequences would be underrepresented in the training mix.

Lack of long-context instruction tuning data. A subtler gap the paper identifies is the scarcity of long-context instruction data for the instruction tuning phase. While short-context instruction datasets are widely available (CommitPackFT, MathInstruct, etc.), there are far fewer datasets of instruction-response pairs that exercise long-context reasoning over code. The authors solve this through synthetic data generation bootstrapped from the pretraining corpus (Section 2.2), using their own short-context Granite-8b-Code-Instruct-4K model to generate multi-turn instruction data from the repository-level packed documents. This avoids dependence on an existing long-context model (which might not be available open-source) and allows the instruction tuning data to be tailored specifically to the Granite model family's pretraining distribution.

Short-context performance degradation. A persistent concern in long-context extension is that optimizing for long sequences will degrade the model's performance on the short-context tasks it was originally good at. This is a form of catastrophic forgetting — the model "unlearns" its short-context skills during the extended training. The paper's baseline results (Table 5) show this is a real risk: the base models do show slight degradation on HumanEvalPack after extension (e.g., 8B base drops from 43.1% to 40.2% synthesis pass@1). However, the paper demonstrates that the degradation is modest (~1% on average) and that for instruct models, the addition of diverse instruction tuning data actually improves short-context performance slightly, more than offsetting any base-model regression. This addresses a concern that has held back long-context extension in practice — the fear that context extension is a zero-sum trade-off where long-context gains come at the expense of short-context quality.

How This Paper Positions Itself

The paper positions itself as a practical recipe for context-length extension that synthesizes several known techniques — RoPE frequency adjustment, sequence parallelism, lightweight continual pretraining — with novel data engineering innovations (repository-level file packing with dependency-aware ordering, per-language length upsampling) and a bootstrapped instruction tuning pipeline. The core hypothesis, which the authors attribute to Fu et al. (2024), is stated explicitly:

"We hold the basic hypothesis that the ability to utilize information at arbitrary input locations is a capability that is mostly already acquired through large-scale pretraining, and that this capability can be readily extended to contexts substantially longer than seen during original pretraining (e.g., 4K to 128K) through lightweight training on appropriate data mixture."

This framing is important because it rejects the premise that long-context capability requires fundamentally different model architecture or training from scratch. Instead, the paper argues that the base model already "knows how" to use positional information for attention — the RoPE mechanism generalizes naturally to unseen positions — and that the missing ingredient is simply exposing the model to training data that forces it to exercise this capability at extended lengths. The "lightweight training on appropriate data mixture" is doing two things: (1) adjusting the RoPE base frequency so that the positional embeddings for positions beyond the original training length remain well-distributed and distinguishable (preventing them from collapsing or aliasing), and (2) providing enough long-sequence training examples for the model to learn the specific cross-file dependency patterns that characterize repository-scale code understanding.

The paper's contribution is thus not a single novel technique but rather an integrated, validated pipeline for extending open-source code models from short-context to long-context, with concrete engineering decisions (RoPE theta values per context stage: 100K → 250K → 500K → 2M → 10M, batch size 32, 500 steps per stage), a specific data preparation methodology (DAG-based dependency ordering, depth-first folder traversal for unconnected files, 10% downsampling rate for documents under 4096 tokens), and a synthetic instruction data generation recipe. By releasing the resulting models under Apache 2.0, the paper provides not just a method description but a reference implementation that the community can build on directly — a pragmatic stance that addresses the stated gap between proprietary long-context capabilities and what is available in open source.

3. Technical Approach

3.1 Reader Orientation

The paper builds a long-context code language model — specifically, it takes existing Granite 3B and 8B code models that were originally trained with 2K and 4K token context windows respectively, and extends their effective context length to 128K tokens without retraining from scratch. The system solves the problem of enabling repository-scale code understanding on models that were originally restricted to seeing only one or two source files at a time, using a two-phase approach: first, a lightweight continual pretraining phase that teaches the model to attend across much longer sequences by adjusting its positional encoding and reorganizing its training data into repository-level structures, and second, an instruction tuning phase that adds the ability to follow natural language instructions across these long contexts by generating synthetic multi-turn training examples bootstrapped from the pretraining corpus.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a sequential pipeline:

  1. Data Preparation Engine — takes raw code repositories and transforms them into long-context training sequences by packing files from the same repository together in a semantically meaningful order. This component handles dependency analysis (building a directed acyclic graph from file imports), cycle breaking, topological sorting, and per-language length-based upsampling to ensure long sequences are adequately represented in the training distribution.

  2. Continual Pretraining Pipeline — takes the existing short-context Granite base models and fine-tunes them through a progressive schedule where the context window doubles at each stage (8K → 16K → 32K → 64K → 128K), with the RoPE base frequency ($\theta$) adjusted at each stage to support the longer range. This phase uses full attention (no sparse or linear approximations) with sequence parallelism and Flash Attention 2, switching to Ring Attention at the 128K stage.

  3. Synthetic Instruction Data Generator — bootstraps long-context instruction tuning data from the repository-level packed documents by using the original Granite-8b-Code-Instruct-4K model to generate multi-turn interactions. It parses documents into classes, methods, and stand-alone functions, then generates instructions for retrieval, explanation, and implementation tasks that require reasoning across the full packed context.

  4. Instruction Tuning Pipeline — fine-tunes the long-context base models on a mixture of permissively licensed short-context instruction data (CommitPackFT, MathInstruct, Glaive datasets, etc.) and the synthetically generated long-context instruction data, using a multi-turn loss mask with EOS tokens appended after each assistant response to prevent runaway generation.

Information flows as follows: raw repositories enter the Data Preparation Engine → repository-level packed sequences are produced → these sequences feed the Continual Pretraining Pipeline, which progressively extends the base model's context window → the same packed documents are fed to the Synthetic Instruction Data Generator, which produces multi-turn instruction-response pairs → these pairs are combined with short-context instruction datasets → the combined instruction data fine-tunes the long-context base model through the Instruction Tuning Pipeline → the final long-context instruct model emerges.

3.3 Roadmap for the Deep Dive

  • First, the data preparation methodology — repository-level file packing with dependency-aware ordering, the directed acyclic graph construction, and per-language length upsampling — because the quality of the training data is the primary enabler of everything that follows; without properly structured long sequences, the model has nothing meaningful to learn from.

  • Second, the RoPE base frequency adjustment and the progressive training schedule — because this is the core mechanism that physically enables the model to attend to positions beyond its original training length, and understanding the relationship between RoPE theta and context length is necessary to follow the progressive doubling strategy.

  • Third, the continual pretraining logistics — the specific hyperparameters, hardware parallelism strategies (sequence parallelism, Flash Attention 2, Ring Attention), and training budget (500 steps per stage, 4B total additional tokens) — because these practical engineering decisions determine whether the theoretical approach is actually executable at 128K scale.

  • Fourth, the synthetic long-context instruction data generation pipeline — because this is a novel approach to solving the scarcity of long-context instruction data by bootstrapping from the pretraining corpus itself, and the specific task types (retrieval, explanation, implementation) reveal what capabilities the authors believe matter most for repository-scale code understanding.

  • Fifth, the instruction tuning mixture and training configuration — the combination of short and long-context data, the multi-turn loss masking strategy, and the specific datasets used — because this is where the paper's design choice to preserve short-context performance while adding long-context capability is operationalized.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical engineering paper whose core idea is that a short-context code model can be extended to 128K tokens through a combination of (1) reorganizing training data into repository-level packed sequences with dependency-aware ordering, (2) progressively increasing the RoPE base frequency to support longer positional ranges, and (3) synthesizing long-context instruction data from the pretraining corpus to avoid dependence on external long-context models.

Repository-Level File Packing with Dependency-Aware Ordering

The fundamental data engineering challenge the paper tackles is: how do you construct training sequences that are both long enough to teach the model to use a 128K context window, and semantically coherent enough that the long-range attention being practiced is actually useful for downstream code tasks? The naive approach — concatenating code files randomly until a target length is reached — produces sequences where adjacent files may come from entirely different repositories, programming languages, and problem domains. The model might learn to attend across long distances, but the attention patterns it develops would be arbitrary rather than reflecting the genuine cross-file dependencies that characterize real software development.

The paper's solution is repository-level file packing with a dependency-aware ordering algorithm. This is a data preprocessing pipeline that takes a code repository as input and produces a single long document as output, where the files from that repository are concatenated in an order that respects and exploits semantic dependencies. The algorithm operates in several stages:

Stage 1: Dependency graph construction. The algorithm analyzes file imports to build a directed acyclic graph (DAG) where each node represents a file in the repository and each directed edge represents an API import relationship — file A imports functionality from file B, so file B is a dependency of file A. The edge direction points from the dependent file to its dependency. This graph captures the "if you want to understand file A, you should probably read file B first" relationships that a human developer would naturally follow when exploring a codebase.

Stage 2: Cycle breaking. Real-world code repositories often contain circular dependencies — file A imports from file B, and file B imports from file A — which create cycles in the dependency graph and prevent topological sorting. The paper states that the algorithm breaks these cycles to produce a DAG, though the specific cycle-breaking strategy (which edges are removed, whether it is heuristic or random) is not detailed in the text. This is a practical necessity: cycles reflect genuinely messy dependency structures in real codebases, and any ordering algorithm must handle them gracefully.

Stage 3: Topological sorting. Once the graph is acyclic, the algorithm performs a topological sort — a linear ordering of nodes such that for every directed edge from node A to node B, node B comes before node A in the ordering. In the code dependency context, this means: if file X depends on file Y (the code in X imports or calls functions defined in Y), then file Y appears before file X in the final document. The model reads Y first, then X, which means when it reaches the code in X that references Y's functionality, it has already seen Y's definitions and can potentially learn to attend back to them.

Stage 4: Final ordering with structural heuristics. The topological sort provides a partial order, but repositories contain files that are not connected by import relationships — utility scripts, configuration files, documentation, test files that exist alongside but do not import from the main source. The paper addresses this with a layered ordering strategy:

"We then organize the files in a repository by placing documentation and build files first, followed by the ordered set of files with semantic dependencies, and finally the remaining non-connected files. These non-connected files are arranged according to their folder structure, using a depth-first search to traverse the repository."

This creates a document structure with three zones: (1) a header zone containing high-level project information (READMEs, build configurations, documentation), (2) a core zone containing the dependency-ordered source files where the topological sort applies, and (3) a tail zone containing non-connected files arranged by their position in the file system tree (depth-first search through folders). The model experiences a coherent project narrative: first it sees what the project is about and how it is built, then it reads the source code in dependency order (dependencies before dependents), and finally it encounters the remaining files in their natural directory organization.

Stage 5: Per-language organization. After packing repositories into documents, the algorithm determines the dominant programming language of each repository based on file extensions and the presence of build files, and organizes documents by language. This means that during training, the model will see blocks of documents in the same language, which allows it to develop language-specific long-range attention patterns — Python cross-file dependencies look different from Java cross-file dependencies, and grouping by language prevents the model from having to constantly context-switch between different import semantics and module systems.

The paper reports that after processing, the training corpus contains 173,336 documents with a mean length of 73,451 tokens. This is the key statistic: the average document is already above 73K tokens, meaning the majority of the training data exercises context lengths in the 64K–128K range, which is exactly what the model needs to practice for the target 128K window.

Per-Language Context Length Upsampling

A subtle but critical data engineering decision: the distribution of document lengths in the training corpus is not uniform across programming languages. The paper notes that "long data usually come from particular sources" — some languages or project types naturally produce larger repositories with more files, while others tend toward smaller, more compact codebases. If the training data is used as-is, languages with naturally shorter repositories would be underrepresented in the long-context training signal, and the model would learn long-range attention patterns that are biased toward the languages and project structures that happen to produce long documents.

To address this, the paper implements per-language context length upsampling, a technique also used in Fu et al. (2024) and Yu (2023). The specific policy:

"We artificially oversampled longer document sequences on a per-language basis to ensure the quantity of long sequences, thereby improving the overall quality of our training data corpus... In particular, we downsample documents under 4096 tokens to a rate of 10%."

This is worth parsing carefully. The term "upsampling longer sequences" means that longer sequences are sampled more frequently during training — if a language has relatively few long documents, those documents are repeated or given higher sampling probability to compensate. The complementary operation is downsampling short documents: documents under 4096 tokens are kept at only 10% of their natural frequency. This means that for every 100 short documents that would appear in a naive uniform sampling, only 10 actually appear in the training stream. The paper states they "find [this] to ensure a sufficient number of total tokens and documents" — the 10% rate is chosen as a balance where short documents are not entirely eliminated (which would lose valuable short-context learning signal) but are sufficiently reduced that long sequences dominate the training distribution.

The 4096-token threshold is significant: it corresponds to the original Granite-8B model's training context length (4K). Documents shorter than this threshold are "within distribution" for the original model — the model already has substantial experience with sequences of this length from its original pretraining. The long-context extension should de-emphasize these familiar sequence lengths and emphasize the longer sequences where the model needs to develop new capabilities.

RoPE Base Frequency Adjustment and Progressive Context Extension

The paper's approach to physically enabling the model to attend to 128K tokens relies entirely on adjusting the RoPE (Rotary Position Embedding) base frequency, a technique introduced by Xiong et al. (2023). To understand why this works, we need to understand what RoPE does and why the base frequency matters.

RoPE encodes position information by applying a rotation to the query and key vectors in the attention mechanism. For a token at position $p$, the rotation angle for the $i$-th dimension pair is:

θi=pbase2i/d\theta_i = \frac{p}{\text{base}^{2i/d}}

where $d$ is the head dimension, $i$ indexes dimension pairs from 0 to $d/2 - 1$, and $\text{base}$ is the RoPE base frequency — traditionally 10,000 in many transformer implementations.

What this equation computes: for each pair of adjacent dimensions in the query/key vectors, RoPE applies a rotation whose angle depends on both the token's position $p$ and the dimension index $i$. Lower dimension indices get larger rotations (more sensitive to position changes); higher dimension indices get smaller rotations (less sensitive). The $\text{base}$ parameter controls how quickly the rotation angles vary with position — a larger base means smaller rotation angles for the same position, which means the positional encoding can distinguish positions over a longer range before the angles wrap around or become indistinguishable.

Why adjusting the base matters for context extension: the original Granite models were trained with a base frequency appropriate for their 2K/4K context windows. At those lengths, the RoPE angles for positions 0 through 2048 (or 4096) are well-separated and the model learns attention patterns that rely on these specific angle relationships. If you simply feed the model a 128K sequence without changing the base frequency, the positional angles for positions beyond 4K will be values the model has never seen during training — they will be "out of distribution" for the learned attention patterns, and the model will fail to attend effectively to the later parts of the sequence. By increasing the base frequency, the angular range that was previously used for 0–4K is "stretched" to cover 0–128K, keeping the per-position angles within a range the model has some familiarity with, even if the specific position-to-angle mapping has changed.

The paper adopts a progressive training approach where the context window is doubled at each stage, and a new optimal RoPE base frequency is found for each stage. The specific schedule:

Context WindowRoPE Theta ($\theta$)
8K100,000
16K250,000
32K500,000
64K2,000,000
128K10,000,000

The progression is not strictly linear or exponential. From 100K to 10M represents a 100× increase in the base frequency, but the jumps are uneven — the 64K → 128K transition (2M → 10M) is a 5× increase, while the 32K → 64K transition (500K → 2M) is a 4× increase. The authors state they "search for the optimal RoPE theta" at each stage, implying that these values are empirically determined through hyperparameter search rather than derived from a theoretical formula. This is consistent with the Xiong et al. (2023) approach, where the optimal base frequency depends on the specific model architecture, training data distribution, and target context length.

At each stage, the model is trained for 500 steps with a batch size of 32. The batch size is fixed across all stages, which means the number of tokens per stage varies: at 8K context, 500 steps × 32 sequences × 8K tokens = 128M tokens; at 128K context, 500 steps × 32 sequences × 128K tokens = 2.048B tokens. The paper reports that the final models are trained for "an extra 4B tokens which is only 0.1% of original pretraining data." This 4B figure is the sum of tokens across all stages of the progressive schedule, and the 0.1% comparison underscores how lightweight this continual pretraining is relative to the original pretraining investment.

A critical design choice: the paper uses full attention throughout this process, explicitly rejecting sparse or linear attention approximations:

"We continue pretrain the full attention Granite code base models using sequence parallelism... without using any sparse or linear attention."

This means the model computes attention between every pair of tokens in the 128K sequence — an $\mathcal{O}(n^2)$ operation with $n = 131,072$. This is computationally expensive but preserves the exact attention mechanism from the original short-context model, avoiding the architectural mismatch that can cause performance degradation when switching attention patterns between training and inference. The model learns attention patterns under the same mathematical operation it will use at inference time, which eliminates a whole category of train-inference discrepancy.

The computational feasibility of full attention at 128K is achieved through two complementary parallelism strategies. For context windows up to 64K, the paper uses Flash Attention 2 with data parallelism. Flash Attention 2 is a memory-efficient exact attention algorithm that reduces the $\mathcal{O}(n^2)$ memory footprint of standard attention by computing attention in tiles that avoid materializing the full attention matrix in GPU high-bandwidth memory. At 128K, this is no longer sufficient, so the paper switches to Ring Attention (Liu et al., 2023a), which distributes the sequence across multiple GPUs arranged in a logical ring, with each GPU computing attention for its local sequence chunk while passing key-value blocks to neighboring GPUs. The paper also uses sequence parallelism (Li et al., 2021), which partitions the sequence dimension across GPUs for the non-attention parts of the transformer (layer normalization, feed-forward networks), enabling the full 128K sequence to be processed despite individual GPU memory constraints.

Continual Pretraining Data and Hyperparameters

The continual pretraining phase uses the same base pretraining data as the original Granite code models (Mishra et al., 2024), but with the repository-level packing and length upsampling transformations applied. The training is focused on a curated set of programming languages:

"This continued training stage focused on a curated selection of programming languages, such as Python, C, C++, Go, Java, JavaScript, and TypeScript, as in Pinnaparaju et al. (2024)."

This language selection covers the major languages used in the evaluation benchmarks (HumanEvalPack tests Python, JavaScript, Java, Go, C++, Rust; RepoQA tests Python, C++, Java, TypeScript, Rust) and represents a practical subset that balances broad coverage with training efficiency. Rust appears in the evaluations but is not explicitly listed in the training languages, which may contribute to the slightly lower Rust performance observed in the RepoQA results (Table 3 shows Rust at 57% for 3B and 74% for 8B at threshold 0.0, consistently below Python scores).

The training hyperparameters for the continual pretraining phase are:

  • Steps per context stage: 500
  • Batch size: 32
  • Total additional tokens: ~4B

The paper does not explicitly state the optimizer, learning rate, or learning rate schedule for the continual pretraining phase, though these are likely carried over from the original Granite pretraining recipe. The 500 steps per stage is a relatively small number — each stage processes only 500 gradient updates — which reinforces the paper's claim that the model already possesses the necessary attention mechanisms and just needs a "nudge" to generalize them to longer ranges. The 4B total tokens figure is derived from summing across all five stages of the progressive schedule.

Synthetic Long-Context Instruction Data Generation

The instruction tuning phase requires instruction-response pairs that exercise long-context code understanding, but such datasets are scarce in the open-source ecosystem — existing instruction tuning datasets for code (CommitPackFT, Self-OSS-Instruct, etc.) are predominantly short-context. The paper's solution is to generate synthetic long-context instruction data by bootstrapping from the pretraining corpus itself, using the original short-context Granite-8b-Code-Instruct-4K model as the generator.

The generation pipeline operates on the same repository-level file-packed documents produced by the data preparation engine. For each packed document, the pipeline creates a multi-turn dataset where instructions target specific capabilities the authors want the long-context model to develop. The paper describes three task types:

Task 1: Retrieval and extraction. The pipeline parses the packed document to identify classes, methods, and stand-alone functions using program analysis (likely AST parsing, though the specific tooling is not named). It then:

"requests and extracts the implementations of a random subset of the extracted functions/methods (up to 5 per file in the document)"

This produces instruction-response pairs where the instruction asks the model to find and reproduce a specific function implementation buried within the long document. The model must learn to locate the relevant function among potentially hundreds of other functions across many files, read its implementation, and output it correctly. The "up to 5 per file" constraint means a typical packed document spanning dozens of files could generate dozens of such retrieval instructions.

Task 2: Explanation with documentation. After extracting implementations, the pipeline:

"asks for an explanation of that implementation using available documentation"

This produces instruction-response pairs where the model must not only find the function but also explain what it does, leveraging any documentation (docstrings, comments, README content) present in the packed context. This task type forces the model to integrate information from potentially distant parts of the document — the function implementation may be in one file while relevant documentation context is in a README at the beginning of the packed sequence.

Task 3: Implementation from context. The pipeline:

"generates instructions for implementing the sampled functions (methods) based on the remaining documentation and code with the function excluded"

This is the inverse of retrieval: the function implementation is removed from the context, and the model must write it from scratch based on documentation, type signatures, usage examples in other files, and the surrounding code structure. This exercises a different kind of long-context reasoning — rather than finding something that exists, the model must synthesize code that is consistent with the project's conventions, API usage patterns, and documentation, all of which are distributed across the packed document.

The responses for these tasks are either parsed semantically from the original document (for retrieval tasks, where the ground-truth answer literally exists in the context) or generated using Granite-8b-Code-Instruct-4K (for explanation and implementation tasks, where the response requires natural language generation or novel code synthesis). Critically, the authors use their own short-context model rather than relying on an external long-context model like GPT-4 or Claude:

"We generate multi-turn instruction data from repository-level file-packed documents with our original Granite-8B-Code-Instruct model to avoid the dependency on an existing long context model."

This is an important practical consideration for open-source model development: if you need a long-context model to generate training data for your long-context model, you have a circular dependency. By using their short-context model, the authors demonstrate that the generation can be bootstrapped from models that already exist in the open-source ecosystem. However, there is a subtlety: the short-context model cannot see the entire packed document at once. The paper does not detail how the short-context model processes the document for generation — it likely chunks the document into 4K-token segments and processes them iteratively, generating per-segment responses that are then combined. This means the generated responses may not fully capture cross-file dependencies that span beyond 4K tokens, potentially limiting the quality of the synthetic data for very long-range reasoning tasks.

The instructions themselves are "human-designed for the purpose of enhancing the long-context performance in specific tasks like generation, retrieval and translation." The authors manually craft instruction templates that simulate the kinds of queries a developer would make when working with a repository-scale codebase — "find the implementation of function X," "explain how module Y works using the available documentation," "implement function Z based on the existing codebase." The specific templates are not provided in the paper, but the task descriptions give a clear picture of the intended coverage.

The multi-turn structure means that within a single training example, the model sees a sequence of alternating user instructions and assistant responses, all grounded in the same long context document. This simulates an extended interaction where the developer asks multiple questions about the same codebase — exactly the usage pattern of a long-context coding assistant. The repetition of instructions for different functions continues "until the desired length was achieved," meaning the multi-turn sequences are themselves long, pushing the model to maintain coherent attention across many turns of dialogue while keeping the full repository context available.

Instruction Tuning Data Mixture and Training Configuration

The instruction tuning phase fine-tunes the long-context base models on a mixture of short and long-context instruction data. The paper states the goal explicitly:

"By exposing the model to both short and long context data, we aim to enhance its long context capability without sacrificing code generation performance at short input context."

This is a deliberate design choice that addresses the catastrophic forgetting concern: if you fine-tune only on long-context data, the model may overfit to the long-context task distribution and lose the short-context capabilities that were its original strength. By maintaining a substantial fraction of short-context data, the model is forced to remain proficient at both regimes.

Short-context instruction data components. The paper lists the following datasets, all permissively licensed:

  • CommitPackFT (Muennighoff et al., 2023) — a dataset of commit messages paired with code diffs, teaching the model to understand and generate code changes.
  • MathInstruct (Yue et al., 2023) — a dataset of mathematical reasoning instructions, with the paper noting that "GSM8K-RFT and Camel-Math" subsets were removed due to "unknown or NC license" — demonstrating attention to licensing compliance.
  • MetaMathQA (Yu et al., 2023) — bootstrapped mathematical question-answering data.
  • Glaive-Code-Assistant-v3 — a code-focused instruction dataset from Glaive AI.
  • Self-OSS-Instruct-SC2 — self-instruct-style data for open-source code tasks.
  • Glaive-Function-Calling-v2 — instruction data for function-calling API interactions.
  • NL2SQL — natural language to SQL conversion data.
  • HelpSteer (Wang et al., 2023b) — a multi-attribute helpfulness dataset.
  • OpenPlatypus (Lee et al., 2023) — a refined instruction dataset.
  • Synthetically generated API calling data (Basu et al., 2024) — API interaction examples.
  • Synthetically generated multi-turn code interactions with execution feedback — multi-turn coding dialogues with execution results.

This is a broad mixture spanning code generation, mathematical reasoning, function calling, and general instruction following. The diversity is intentional: it prevents the model from specializing too narrowly on long-context repository tasks and helps maintain the general coding assistant capabilities of the original Granite instruct models.

Long-context instruction data. The synthetic data described in the previous subsection, with three task types (retrieval, explanation, implementation from context) organized as multi-turn dialogues.

The paper does not specify the mixing ratio between short and long-context data — this is a notable omission, as the ratio is likely an important hyperparameter that trades off long-context capability against short-context preservation. The results in Table 5 suggest the chosen ratio works well (HumanEvalPack degradation is minimal), but the specific value is not reported.

Training configuration. The instruction tuning follows the same training parameters as the original short-context Granite instruct models (Mishra et al., 2024):

  • Global batch size: 128
  • Learning rate: 2 × 10⁻⁵
  • Noise multiplier for input embeddings: 5
  • Padding-free transformers — sequences are packed without padding tokens, using a variant of the sequence packing approach that avoids wasting computation on pad tokens.

The noise multiplier of 5 on input embeddings is an interesting detail: it adds Gaussian noise to the input embeddings during training, scaled by a factor of 5 relative to the standard deviation of the embeddings. This is a form of regularization that can improve robustness to input perturbations and potentially help the model generalize to out-of-distribution token positions (since the long-context positional embeddings are somewhat out-of-distribution relative to the original training). The paper does not elaborate on the motivation for this specific value.

Multi-turn loss masking. The instruction tuning data contains multi-turn dialogues where each sample is a sequence of alternating user and assistant messages. During training, the loss is computed only on the assistant's response tokens, not on the user's instruction tokens. The paper describes the implementation:

"We use a multiturn loss mask for each sample, as in Wang et al. (2023a). This is particularly important as our finetuning data corpus consists of instruction-response pairs with multiple turns."

The loss mask is a binary vector of the same length as the input sequence, where positions corresponding to user instruction tokens have mask value 0 (no loss computed) and positions corresponding to assistant response tokens have mask value 1 (loss computed). This ensures the model learns to generate helpful responses but does not learn to predict user instructions — the user's turns are treated as conditioning context, not as targets.

EOS token handling. An important inference-time consideration:

"When composing a sequence, we append an EOS token after each response from the model to prevent runaway generation during inference."

During training, each assistant turn in the multi-turn dialogue is terminated with an End-of-Sequence token. This teaches the model to stop generating after completing a response rather than continuing indefinitely. Without this, a model trained on multi-turn dialogues might learn to generate the next user turn after finishing its own response (since that is what it sees in the training data), leading to "runaway generation" where the model hallucinates an entire multi-turn conversation rather than stopping and returning control to the user.

Sequence Parallelism and Attention Implementation Details

The computational infrastructure for training at 128K context deserves attention because it is what makes the approach practically feasible. The paper uses a layered parallelism strategy:

Sequence parallelism (Li et al., 2021) for the non-attention transformer components:

"We continue pretrain the full attention Granite code base models using sequence parallelism"

In standard data parallelism, each GPU holds a full copy of the model parameters and processes a different batch of data — but for extremely long sequences (128K tokens), a single sequence may not fit on one GPU even if the model parameters do. Sequence parallelism addresses this by splitting the sequence dimension across GPUs: GPU 0 holds tokens 1–32K, GPU 1 holds tokens 32K–64K, and so on. For operations that are independent across sequence positions (layer normalization, feed-forward networks, embedding lookups), each GPU processes its chunk independently. For operations that mix information across positions (attention), the GPUs must communicate to share key-value pairs. The paper uses the EasyContext implementation (referenced via GitHub URL in Section 2.1).

Flash Attention 2 for contexts up to 64K. Flash Attention is an exact attention algorithm that computes attention outputs without ever materializing the full $N \times N$ attention matrix in GPU high-bandwidth memory. It works by tiling the computation: the query matrix is split into blocks, the key-value matrices are split into blocks, and attention is computed block-by-block, with intermediate softmax statistics carefully tracked and recombined. This reduces the memory complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(N)$ while producing mathematically identical outputs to standard attention. Flash Attention 2 is an optimized version with better parallelism and reduced non-matmul FLOPs.

Ring Attention (Liu et al., 2023a) for the 128K stage:

"We train with data parallelism and Flash Attention 2 until 64K tokens and then used Ring Attention to reach 128K tokens."

Ring Attention extends the blockwise attention concept to multi-GPU settings. GPUs are arranged in a logical ring. Each GPU holds a chunk of the query sequence and computes attention against its local key-value chunk. It then passes its key-value chunk to the next GPU in the ring while receiving the previous GPU's key-value chunk. By circulating key-value chunks around the ring, each GPU eventually computes attention against all key-value positions. This communication pattern is efficient because it overlaps communication with computation — while one key-value block is being transferred, the GPU computes attention against a previously received block. The ring topology minimizes the number of communication hops compared to all-to-all communication.

The transition point from Flash Attention 2 to Ring Attention at 64K → 128K reflects a practical memory threshold: at some sequence length, even Flash Attention's memory-efficient exact attention exceeds a single GPU's memory, and the distributed approach of Ring Attention becomes necessary.

Summary of Design Choices and Their Justifications

  • Progressive context doubling with per-stage RoPE theta search over a single jump to 128K: allows the model to gradually adapt its attention patterns; empirical evidence from Xiong et al. (2023) supports progressive schedules over one-shot extension; the optimal RoPE theta depends on sequence length, so a single theta for all stages would be suboptimal.

  • Repository-level file packing with DAG-based ordering over random concatenation or per-file training: preserves semantic dependencies between files, teaching the model to attend across files in the order a human developer would naturally read a codebase; the DAG captures import relationships that are the primary mechanism of cross-file coupling in real code.

  • Depth-first folder traversal for non-connected files over alphabetical or random ordering: preserves the hierarchical organization that developers use to navigate projects; depth-first search means sibling files in the same directory stay together, which is useful because files in the same directory often share thematic relationships even without explicit imports.

  • Per-language length upsampling with 10% short-document retention over uniform sampling: ensures all languages contribute sufficiently long sequences to the training mix; the 10% retention rate preserves some short-context signal to avoid catastrophic forgetting of within-file code generation patterns.

  • Full attention (no sparse or linear approximations) over architectural modifications: maintains exact parity with the short-context model's attention mechanism, eliminating train-inference discrepancy; the computational cost is paid through parallelism (Flash Attention 2, Ring Attention) rather than architectural approximation.

  • Synthetic instruction data bootstrapped from pretraining corpus over external long-context model generation: avoids dependency on proprietary models or existing long-context models that may not exist for the target domain; ensures the instruction data distribution matches the pretraining data distribution; the "retrieval, explanation, implementation from context" task types directly exercise the skills evaluated by RepoQA, LCC, and RepoBench-P.

  • Multi-turn loss masking with EOS tokens over full-sequence loss or no EOS tokens: the loss masking prevents the model from learning to predict user turns; the EOS tokens teach the model to stop generation after completing a response, preventing the hallucinated multi-turn conversations that plague multi-turn fine-tuned models.

  • Mix of short and long-context instruction data over long-context-only fine-tuning: prevents catastrophic forgetting of short-context code generation capabilities; the diverse short-context datasets (math, SQL, function calling, general instruction following) maintain the model's breadth as a general coding assistant.

4. Key Insights and Innovations

Innovation 1: Context-Length Extension as a Data Distribution Problem, Not a Positional Encoding Problem

The dominant framing in long-context LLM research has been that the primary obstacle to longer context windows is the positional encoding — specifically, that RoPE embeddings for positions beyond the original training length are "out of distribution" and the model cannot generalize to them. The solution in prior work (Xiong et al., 2023) focused almost entirely on adjusting the RoPE base frequency to make these extended positions numerically well-behaved, often treating the training data as a secondary concern.

This paper makes a subtle but intellectually significant pivot: the model already knows how to attend across arbitrary positions — it just hasn't been given anything meaningful to attend across. The authors' central hypothesis, drawn from Fu et al. (2024), states that "the ability to utilize information at arbitrary input locations is a capability that is mostly already acquired through large-scale pretraining." The missing ingredient is not a better positional encoding formula but training data that exercises long-range dependencies in a semantically coherent way.

This reframing has concrete intellectual consequences. It explains why simply cranking up the RoPE theta without corresponding data engineering produces models that can attend to long contexts in principle but don't use that capacity effectively — they get uninformative training signal from randomly concatenated files that share no meaningful relationships. The paper's repository-level file packing with dependency-aware ordering is not just a preprocessing trick; it embodies the insight that long-context capability emerges from practicing the specific cross-file reasoning patterns that characterize real software development — import chains, API usage across modules, project-wide conventions. A model trained on dependency-ordered repositories learns that attending to file A when reading file B is useful because file B imports from file A, and this causal structure is present in the training data by construction.

The evidence for this reframing is implicit but powerful: the approach works with only 4B additional tokens (0.1% of original pretraining), which would be inexplicable if the model were learning fundamentally new attention mechanisms from scratch. Instead, the tiny training budget suggests the model is simply being shown where to direct the attention capabilities it already possesses. The dramatic performance gaps on RepoBench-P (Table 2: +36.7 percentage points for 8B at 32K) and RepoQA (Table 3: +38.6 percentage points for 8B at threshold 0.8) with negligible short-context degradation (Table 5: ~1% average drop) are consistent with this interpretation — the model isn't trading off short-context for long-context ability; it's adding a new skill on top of preserved existing ones, which is what you'd expect if the underlying mechanism (attention) remains unchanged and only the target patterns (where to attend) are being refined.

This is a fundamental reframing, not an incremental improvement. It shifts the research question from "how do we design positional encodings that work at 128K?" toward "how do we construct training data that teaches models what cross-file dependencies look like?" — a question with different research methods (program analysis, dependency graphs, repository structure mining) than the architecture-focused approach that previously dominated.

Innovation 2: Dependency-Aware Repository Packing as a General-Purpose Data Engineering Primitive

Prior approaches to creating long-context code training data used one of two strategies: either random concatenation of files until a target length is reached (simple but destroys repository structure) or per-file training with file-level metadata appended (preserves file identity but loses multi-file coherence). Both strategies treat code files as independent documents rather than as components of a larger dependency structure.

The paper introduces dependency-aware repository packing — a data engineering primitive that reorganizes code repositories into linear sequences where the ordering respects import dependencies — and demonstrates that this alone (combined with RoPE adjustment) is sufficient to unlock repository-scale understanding. The specific mechanism (DAG construction from imports, cycle breaking, topological sorting, depth-first folder traversal for unconnected files) is described in Section 3, but the intellectual contribution is the recognition that the order in which the model encounters files during training fundamentally shapes what kind of cross-file reasoning it learns.

This is significant beyond the immediate performance gains because it is a transferable design pattern. The insight is not specific to code — any domain with structured dependencies between documents (legal corpora with citation networks, scientific literature with reference chains, documentation sets with cross-references) could apply analogous dependency-aware packing to teach models the relevant multi-document reasoning patterns. The paper's DAG construction from imports is one instantiation of a more general idea: structure your training sequences so that the forward pass through the model mirrors the reasoning path a human would take through the material. A developer reading a repository reads dependencies before dependents; a lawyer reading case law reads the cited precedent before the citing opinion; a researcher reads the foundational paper before the follow-up work. By encoding these natural reading orders into the training data, the model learns attention patterns that align with human reasoning.

The evidence for this innovation's distinctiveness comes from comparing the paper's results to what would be expected from random concatenation. If ordering didn't matter, then simply packing files from the same repository (without dependency ordering) would perform similarly — but the paper specifically calls out the ordering algorithm as "critical for long-context continual pretraining" (Section 2.1), implying that ablation experiments (not shown but referenced in the design rationale) demonstrated its importance. The mean document length of 73,451 tokens after processing means the typical training example spans an entire repository with meaningful internal structure, not a bag of unrelated files.

This is a moderate innovation — it combines known techniques (DAG topological sorting, depth-first traversal) in a novel combination applied to a new problem — but one with outsized practical impact because it provides a concrete, implementable recipe that the open-source community can adopt immediately. The per-language length upsampling (Section 3.2) is a complementary insight that addresses a distribution-shift problem: "long data usually come from particular sources," and without explicit rebalancing, the model's long-context training signal is biased toward the languages and project structures that happen to produce large repositories. This is a diagnostic contribution — it identifies a confounding factor (language is correlated with document length) that would silently degrade performance if not addressed — and provides a simple correction (per-language oversampling of long sequences, downsampling of short sequences to 10% retention).

Innovation 3: Bootstrapping Long-Context Instruction Data Without External Long-Context Models

A chicken-and-egg problem plagues long-context instruction tuning: to fine-tune a model to follow instructions over long contexts, you need instruction-response pairs that span long contexts, but generating such data typically requires a model that can already handle long contexts (to produce coherent responses grounded in the full document). This creates a circular dependency — you need a long-context model to produce training data for a long-context model. Prior work often sidesteps this by using proprietary long-context models (GPT-4, Claude) to generate synthetic data, but this violates the open-source ethos and introduces distribution-shift issues.

The paper's solution is bootstrapping from the short-context model and the pretraining corpus itself. The synthetic instruction data is generated by the original Granite-8b-Code-Instruct-4K model — a model that cannot see the full packed document at once — operating on repository-level documents parsed into constituent functions, methods, and classes. The key insight is that not all long-context instruction data requires the generation model to have long-context capability. For retrieval tasks (Task 1 in Section 3.4), the ground-truth answer literally exists in the document and can be extracted programmatically without any model generation. For explanation tasks (Task 2), the model needs to produce natural language, but the content to explain can be localized to individual functions. For implementation tasks (Task 3), the context needed to write a function (its signature, documentation, usage examples in other files) can often be collected into a window that fits the short-context model's capacity.

This is a conceptual contribution about data generation methodology: it reframes the problem from "generate instruction data using a long-context model" to "decompose the long-context document into instruction-target pairs that can be generated with short-context tools." The multi-turn structure (repeated instructions targeting different functions until the desired sequence length is achieved) means the generated training example is long even though each individual generation step was short-context. The model learns to maintain attention across many turns of dialogue, each of which exercises a different sub-region of the full context — exactly the usage pattern of a developer asking multiple questions about the same codebase in sequence.

The evidence that this bootstrapping works effectively comes from the instruct model results. Table 4 shows Granite-8b-Code-Instruct-128K achieving 61.6% retrieval accuracy on RepoQA at threshold 0.5, compared to 0.6% for the 4K version — a 100× improvement. Figure 1 visualizes this gap dramatically: the short-context instruct models are effectively at zero across all five programming languages at threshold 0.5, while the long-context versions achieve 38–73% (language-dependent). If the synthetic data were low-quality (because the generating model couldn't see cross-file dependencies), these retrieval and understanding tasks would fail — but they succeed, indicating that the decomposition strategy produced training signal that generalizes to genuine long-context reasoning.

The significance beyond this paper is that the bootstrapping approach eliminates a barrier to entry for open-source long-context model development. Any team with a pretrained short-context model and a code corpus can generate long-context instruction data without access to proprietary long-context models. This is a practical innovation that democratizes long-context instruction tuning.

Innovation 4: Full Attention Preservation as a Principle for Context Extension Without Regret

A persistent trade-off in long-context modeling is that enabling longer sequences requires architectural compromises — sparse attention patterns, linear attention approximations, sliding windows, or state-space models — that reduce the computational burden but often degrade performance on the short-context tasks the model was originally designed for. The field has largely accepted this as an unavoidable tension: you can have long context or you can have preserved short-context quality, but not both at full strength.

This paper challenges that assumption by demonstrating that you can extend context 32–64× (from 4K to 128K) with full, exact attention throughout, preserving short-context performance to within ~1% of the original model, without architectural modification. The key is that the computational burden of full attention at 128K is handled through parallelism strategies (Flash Attention 2, Ring Attention, sequence parallelism) rather than through attention approximation — the mathematical operation the model performs at training and inference is identical to the short-context version, just distributed across more hardware.

This is an architectural principle, not just an implementation detail. It says: when extending context length, preserve the attention mechanism exactly, and pay the computational cost through engineering rather than approximation. The benefit is that there is no train-inference gap, no mismatch between the attention patterns learned during pretraining and those used at inference, and no question about whether a degradation in short-context performance is due to the context extension or due to the architectural compromise. The paper's results in Table 5 bear this out: Granite-8b-Code-Base-128K scores 40.2% synthesis pass@1 on HumanEvalPack versus 43.1% for the 4K base model — a 2.9 percentage point drop — which is remarkably small for a 32× context extension. The instruct model actually improves from 49.6% to 51.4%, suggesting that when the instruction tuning data mixture is well-designed, the long-context training can even provide positive transfer to short-context tasks.

The significance of this principle extends beyond this paper's specific implementation. It provides a clear decision criterion for practitioners: if you have the hardware to parallelize full attention at your target context length (through Flash Attention, Ring Attention, or future efficient attention implementations), prefer preserving exact attention over architectural approximation. The 0.1% pretraining budget needed for the extension (4B out of ~4T original tokens, assuming typical code model pretraining scales) means the approach is economically viable even without the efficiency gains of sparse attention — the dominant cost remains the original pretraining, and the context extension adds negligible overhead.

This is a moderate innovation — the parallelism techniques (Flash Attention 2, Ring Attention) are not novel, and the principle of preserving full attention was implicit in prior work — but the paper's explicit articulation of the principle, combined with the empirical demonstration that it works at 128K scale with minimal short-context degradation, makes it a useful contribution to the practitioner's decision framework. It removes "architectural compromise" from the list of necessary evils in context extension and replaces it with "engineering investment," which is a trade-off many teams are better positioned to make.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on multiple benchmarks spanning both short and long-context code tasks. For long-context evaluation, the primary datasets are Long Code Completion (LCC) (Guo et al., 2023), which tests next-line code prediction from repository-level context for Python, Java, and C#; RepoBench-P (Liu et al., 2023c), which also tests next-line completion using the Cross-File-First subset; RepoQA (Liu et al., 2024), a needle-in-the-haystack retrieval benchmark where the model must find a specific function in 16K-token context given a natural language description, spanning 5 languages × 10 repositories = 500 subtasks; and Key Retrieval, a synthetic benchmark built on the Code Contest finetuning dataset (Li et al., 2022) where the model must execute a key Python function buried within concatenated Python solutions at varying sequence lengths and offsets. For short-context evaluation, the paper uses HumanEvalPack (Muennighoff et al., 2023), which extends HumanEval's Python problems to five additional languages (JavaScript, Java, Go, C++, Rust) and tests three coding tasks: synthesis, explanation, and fixing. The specific splits used are not detailed in the paper beyond the descriptions in Section 3.1; RepoQA uses 500 subtask tests, and LCC and RepoBench-P are rebalanced for equal representation across context-length buckets (each bucket has 100 samples when possible).

  • Base model(s). The experiments start from the Granite 3B and 8B code models (Mishra et al., 2024), which were originally pretrained with context windows of 2K and 4K tokens respectively. These models are described as "full attention" transformers using RoPE positional encoding. The 3B and 8B sizes were chosen to represent two points on the parameter scale that are practical for open-source deployment — small enough to run on consumer hardware but large enough to demonstrate meaningful code generation and understanding capabilities. The original models serve as the direct short-context baselines throughout the evaluation, enabling a clean before-and-after comparison of the context-extension process.

  • Metrics. The paper reports several task-specific metrics. For LCC and RepoBench-P, the metric is Exact Match (EM) accuracy — whether the model's predicted next line of code matches the ground-truth line exactly — reported at specific context lengths (4K, 8K, 16K, 32K). For RepoQA, the metric is retrieval accuracy at varying similarity thresholds from 0.0 to 1.0, where the threshold represents how closely the model's retrieved output must match the ground-truth function (0.0 being any match, 1.0 being exact match); accuracy is reported per programming language and averaged across the 5 languages. For Key Retrieval, the metric is a pass/fail heatmap showing whether the model correctly executes the key function across a grid of sequence lengths (x-axis) and key offset positions (y-axis). For HumanEvalPack, the metric is pass@1 using greedy decoding, reported separately for synthesis, fix, and explain tasks, and averaged across all tasks and languages. All evaluations use greedy decoding with a 256 new-token limit for retrieval tasks; HumanEvalPack results are reported as pass@1 percentages.

  • Baselines. The primary baselines are the original short-context Granite models before context extension: Granite-3b-Code-Base-2K, Granite-8b-Code-Base-4K, Granite-3b-Code-Instruct-2K, and Granite-8b-Code-Instruct-4K. These are the direct predecessors to the 128K models, trained by the same team using the same base pretraining recipe but with original context-length constraints. The paper also implicitly compares against the general landscape of open-source code models (CodeGemma, Code Llama) referenced in Section 1, though no direct head-to-head benchmarks with these external models are reported in the paper's tables — the comparison is against the short-context Granite versions rather than against competing model families. This is a notable evaluation design choice: the paper is primarily demonstrating the relative improvement from context extension rather than benchmarking against the state-of-the-art in long-context code modeling.

  • Generation budget / compute accounting. The paper does not conduct a FLOPs-matched comparison or budget-constrained efficiency analysis between long and short-context models — this is not a compute-optimal scaling study. Instead, the evaluation simply compares models at their respective native context lengths. The long-context models are evaluated with sequences up to 128K tokens, while the short-context baselines are evaluated at their original 2K/4K limits, and both are tested at the intermediate context lengths (4K, 8K, 16K, 32K) that appear in the rebalanced LCC and RepoBench-P benchmarks. The "budget" distinction is in training cost: the paper emphasizes that the long-context models were produced with only 4B additional tokens of continual pretraining (0.1% of original pretraining data), but this training-cost metric is not carried through as a controlled variable in the evaluation design.

  • Cross-validation / statistical protocol. The paper does not report any cross-validation, statistical significance testing, confidence intervals, or error bars on the reported metrics. All results are point estimates from single evaluation runs. The LCC and RepoBench-P benchmarks are rebalanced to ensure equal representation across context-length buckets (100 samples per bucket when possible), which partially addresses the original benchmarks' skew toward shorter sequences, but this is a dataset-balancing procedure rather than a statistical protocol. The absence of uncertainty quantification is a notable limitation — with RepoQA's 500 subtasks split across 5 languages (100 per language), and LCC's bucket sizes of ~100 samples, the point estimates could have meaningful variance that is not captured in the reported tables.

Main Quantitative Results

Long Code Completion (LCC) — Base Models

The headline finding for code completion at long contexts is that the 128K base models dramatically outperform their short-context predecessors at all context lengths from 4K to 32K. Table 1 reports Exact Match scores on the balanced LCC benchmark:

For the 3B model, Granite-3b-Code-Base-2K achieves 24.5% EM at 4K context, declining to 15.4% at 8K, 11.4% at 16K, and 10.0% at 32K. The short-context model's performance monotonically degrades as the context window exceeds its training length, which is expected — beyond 2K, the model cannot effectively attend to the full input. In contrast, Granite-3b-Code-Base-128K achieves 54.6% at 4K, 56.8% at 8K, 52.2% at 16K, and 57.8% at 32K. The absolute gaps are enormous: +30.1 percentage points at 4K, +41.4 at 8K, +40.8 at 16K, and +47.8 at 32K. Notably, the long-context 3B model's performance is roughly flat across context lengths — it scores similarly at 4K and 32K — while the short-context model degrades sharply, suggesting the 128K model has genuinely learned to use long contexts rather than simply being more robust to them.

For the 8B model, the pattern is similar but with higher absolute numbers. Granite-8b-Code-Base-4K starts at 41.9% at 4K (within its training distribution) but falls to 23.7% at 8K, 19.1% at 16K, and 15.0% at 32K. Granite-8b-Code-Base-128K achieves 56.5% at 4K, 60.1% at 8K, 51.8% at 16K, and 57.4% at 32K. The absolute gaps are +14.6, +36.4, +32.7, and +42.4 percentage points respectively. The 8B model shows a slight dip at 16K (51.8%) compared to 8K (60.1%) and 32K (57.4%), which may reflect genuine difficulty with the 16K bucket's specific examples or could be noise in the ~100-sample evaluation.

A critical observation: at 4K context — the original training length for the 8B model — the long-context version still substantially outperforms the short-context version (56.5% vs. 41.9%). This +14.6 point gain at the short-context model's own native length is noteworthy. It suggests that the continual pretraining on repository-level packed data, even at 4K-effective context, teaches the model cross-file reasoning patterns that improve code completion within individual files. Alternatively, it could reflect the benefit of the extended training data (repository-level packing, language upsampling) rather than the context extension per se. The paper does not disentangle these effects — there is no ablation comparing a model trained on the same packed data but with original RoPE theta and context length.

RepoBench-P — Base Models

Table 2 presents the RepoBench-P balanced results, which corroborate the LCC findings with a different benchmark and rebalancing methodology. The pattern of short-context model degradation and long-context model stability is replicated:

For the 3B model, Granite-3b-Code-Base-2K scores 22.0%, 17.9%, 15.4%, and 14.0% at 4K, 8K, 16K, and 32K respectively. Granite-3b-Code-Base-128K scores 39.8%, 46.8%, 43.1%, and 45.3%. Absolute gaps: +17.8, +28.9, +27.7, +31.3 percentage points. The 3B long-context model peaks at 8K (46.8%) rather than 32K, with scores relatively flat from 8K–32K.

For the 8B model, Granite-8b-Code-Base-4K scores 27.9%, 23.0%, 15.7%, and 7.8% — a steep decline at 32K to near-random performance. Granite-8b-Code-Base-128K scores 42.7%, 44.0%, 44.8%, and 44.5% — remarkably consistent across all context lengths. Absolute gaps: +14.8, +21.0, +29.1, +36.7 percentage points. The 8B long-context model's scores across the four context lengths are within a 2.1-point range (42.7%–44.8%), suggesting near-perfect robustness to context length within the 4K–32K range tested.

The absolute EM scores on RepoBench-P are lower than on LCC for both models — the 8B long-context model peaks at 44.8% on RepoBench-P versus 60.1% on LCC. This likely reflects RepoBench-P's greater difficulty (cross-file-first subset, rebalanced sampling) or differences in the benchmark construction. The paper does not provide per-language breakdowns for LCC or RepoBench-P, so the aggregate scores may mask language-specific strengths and weaknesses.

RepoQA — Base Models

Table 3 provides the most granular long-context evaluation, showing retrieval accuracy on RepoQA for base models at 11 similarity thresholds from 0.0 (any match accepted) to 1.0 (exact match required), broken down by programming language (Python, C++, Java, TypeScript, Rust) and averaged.

Granite-3b-Code-Base-2K achieves near-zero performance at any threshold above 0.0. At threshold 0.0, the model achieves 6.0% (Python), 6.0% (C++), 4.0% (Java), 7.0% (TypeScript), and 1.5% (Rust), for an average of 4.9%. At all thresholds from 0.1 to 1.0, every language scores 0.0% — the model produces something retrievable (hence nonzero at threshold 0.0) but never produces output that matches the target function at even the loosest similarity threshold. This is a stark demonstration of complete long-context failure: the 2K model cannot locate a specific function in a 16K-token repository context.

Granite-3b-Code-Base-128K shows substantial retrieval capability. At threshold 0.0, average accuracy is 61.6% (Python 76.0%, C++ 58.0%, Java 59.0%, TypeScript 58.0%, Rust 57.0%). Performance degrades gracefully as the similarity threshold increases: 46.2% at 0.1, 42.4% at 0.2, 39.4% at 0.3, 36.4% at 0.4, 33.8% at 0.5, 30.6% at 0.6, 27.8% at 0.7, 26.2% at 0.8, 20.8% at 0.9, and 15.2% at 1.0 (exact match). The absolute gaps versus the 2K model are +56.7 at threshold 0.0 and +15.2 at threshold 1.0. Python consistently outperforms other languages (76.0% at threshold 0.0 vs. 57–59% for others), which the authors attribute to the model's stronger Python capabilities carried over from pretraining.

Granite-8b-Code-Base-4K fares slightly better than the 3B short-context model but still fails badly. At threshold 0.0, average accuracy is 10.0% (Python 9.0%, C++ 10.0%, Java 11.0%, TypeScript 9.0%, Rust 11.0%). At threshold 0.1, accuracy drops to 0.8% average, and from threshold 0.2 onward, most languages score 0–1% with Rust and TypeScript hitting 0.0% at threshold 0.1 already. The short-context 8B model shows nonzero retrieval at very loose thresholds but collapses completely when any meaningful similarity is required.

Granite-8b-Code-Base-128K achieves the best overall RepoQA performance. At threshold 0.0: average 68.0% (Python 85.0%, C++ 60.0%, Java 57.0%, TypeScript 64.0%, Rust 74.0%). At threshold 0.5: average 46.6% (Python 65.0%, C++ 35.0%, Java 39.0%, TypeScript 40.0%, Rust 54.0%). At threshold 0.8: average 39.2% (Python 54.0%, C++ 32.0%, Java 32.0%, TypeScript 35.0%, Rust 43.0%). At threshold 1.0 (exact match): average 26.8% (Python 45.0%, C++ 23.0%, Java 23.0%, TypeScript 12.0%, Rust 31.0%). The absolute gaps versus the 4K model at threshold 0.8 are +38.6 percentage points on average, with the largest relative gains for Python (+53.0 at threshold 0.8, from 1.0% to 54.0%). The paper highlights the threshold-0.8 gap specifically: "+38.6% for Granite 8B model with a matching threshold of 0.8."

An interesting pattern in the per-language breakdown: Python and Rust are the strongest languages for the 8B 128K model at exact match (45.0% and 31.0%), while TypeScript lags substantially (12.0%). This may reflect data availability differences in the pretraining and continual pretraining corpora. The paper notes that Rust is evaluated but is not explicitly listed in the curated training languages (Section 2.1 lists "Python, C, C++, Go, Java, JavaScript, and TypeScript"), yet Rust performs second-best at exact match — this suggests either that Rust was implicitly included in training, that the model transfers cross-language reasoning patterns effectively to Rust, or that the Rust repositories in RepoQA are simply easier.

Key Retrieval — Instruct Models

Figure 2 presents a heatmap visualization of the Key Retrieval benchmark for Granite-8B-Code-Instruct, comparing the 4K and 128K versions. The x-axis represents sequence length (tokens), and the y-axis represents the key offset percentage (how far into the sequence the target function is buried).

For the Granite-8b-Code-Instruct-4K model, the heatmap shows strong retrieval performance (green/dark cells) only in the region where sequence length ≤ 4K — exactly the model's training context length. Beyond 4K tokens, retrieval success drops sharply and becomes essentially random (red/light cells) regardless of where in the sequence the key function is placed. This is a clean demonstration of the "context-length cliff": the model functions normally within its training distribution and fails catastrophically outside it.

For the Granite-8b-Code-Instruct-128K model, the heatmap is described as "a perfect-all-green performance" across the full range of sequence lengths tested (up to 128K tokens) and all key offset percentages. The paper notes: "we tend to view that this level of retrieval is relatively easy for long-context code LLMs." This is an important interpretive caveat — the Key Retrieval task, while synthetically constructed to test long-range attention, may be too simple to discriminate between strong long-context models. The all-green result demonstrates that the model has not lost the ability to attend to specific positions at long range, but it does not provide a graded measure of retrieval quality at different difficulty levels. This contrasts with RepoQA, where the graded similarity thresholds provide a more nuanced picture of retrieval fidelity.

RepoQA — Instruct Models

Table 4 provides the RepoQA results for instruction-tuned models, using the same 11-threshold evaluation as the base models, with per-language and average scores.

Granite-3b-Code-Instruct-2K shows even worse retrieval than its base counterpart. At threshold 0.0, average accuracy is 10.6% (Python 15.0%, C++ 10.0%, Java 8.0%, TypeScript 11.0%, Rust 9.0%). At threshold 0.1, this collapses to 0.2% average, and from 0.2 onward, all languages score 0.0%. The short-context instruct model fails to retrieve functions at any meaningful similarity, and its threshold-0.0 performance (10.6%) is lower than the base model's (4.9%), though both are essentially at floor.

Granite-3b-Code-Instruct-128K achieves strong retrieval: average 68.0% at threshold 0.0, declining to 38.0% at threshold 0.5, 29.8% at threshold 0.8, and 18.8% at threshold 1.0. The absolute gaps versus the 2K instruct model are substantial: +37.8 at threshold 0.5, +29.8 at threshold 0.8. Interestingly, the 3B instruct model slightly outperforms the 3B base model at threshold 0.0 (68.0% vs. 61.6%) and at most thresholds — the instruction tuning appears to improve retrieval capability beyond what the base model alone provides, likely due to the synthetic retrieval-focused instruction data.

Granite-8b-Code-Instruct-4K performs at floor level. At threshold 0.0, average accuracy is 7.0% (Python 3.0%, C++ 10.0%, Java 8.0%, TypeScript 10.0%, Rust 4.0%). At threshold 0.1, average is 1.0%, and beyond 0.1, most cells are 0–1% with occasional nonzero entries (C++ at 2.0% through threshold 0.9). The 4K instruct model is effectively incapable of function retrieval, performing worse than the 4K base model (10.0% at threshold 0.0 vs. 7.0% for instruct).

Granite-8b-Code-Instruct-128K achieves the best retrieval performance across all models. At threshold 0.0: average 82.4% (Python 89.0%, C++ 63.0%, Java 91.0%, TypeScript 86.0%, Rust 83.0%). At threshold 0.5: average 61.6% (Python 73.0%, C++ 37.0%, Java 73.0%, TypeScript 62.0%, Rust 63.0%). At threshold 0.8: average 47.6% (Python 58.0%, C++ 24.0%, Java 63.0%, TypeScript 40.0%, Rust 53.0%). At threshold 1.0 (exact match): average 28.2% (Python 48.0%, C++ 3.0%, Java 39.0%, TypeScript 11.0%, Rust 40.0%). The absolute gaps versus the 4K instruct model at threshold 0.5 are +61.0 percentage points on average.

Figure 1 visualizes the threshold-0.5 results as a grouped bar chart comparing short and long-context instruct models across the five programming languages. The visual dramatizes the gap: the short-context bars (both 3B and 8B) are at or near zero for all languages, while the long-context bars range from ~37% (C++ for 8B) to ~73% (Python and Java for 8B). The 8B 128K model consistently outperforms the 3B 128K model, with the gap widening at higher similarity thresholds.

A striking detail in the 8B instruct results: Java significantly outperforms other languages at threshold 0.8 (63.0%) and threshold 1.0 (39.0%), while C++ collapses at exact match (3.0%). This language-specific variance is extreme — Java's exact-match retrieval is 13× higher than C++'s. The paper does not discuss or attempt to explain this variance, but it may reflect differences in function structure, naming conventions, or documentation quality across the RepoQA repositories for these languages.

Short-Context Evaluations — HumanEvalPack

Table 5 presents the critical short-context preservation results on HumanEvalPack, comparing all model variants on synthesis, fix, and explain tasks across the 6 languages (Python + 5 additional), evaluated with greedy decoding.

For base models, the long-context versions show small but consistent degradation:

  • Granite-3b-Code-Base-2K: synthesis 33.0%, fix 19.5%, explain 22.2%, average 24.9%
  • Granite-3b-Code-Base-128K: synthesis 30.5%, fix 19.9%, explain 22.4%, average 24.2%
    • Overall average drops by 0.7 percentage points (24.9% → 24.2%); synthesis drops by 2.5 points.
  • Granite-8b-Code-Base-4K: synthesis 43.1%, fix 29.1%, explain 25.4%, average 32.5%
  • Granite-8b-Code-Base-128K: synthesis 40.2%, fix 25.2%, explain 28.2%, average 31.2%
    • Overall average drops by 1.3 percentage points (32.5% → 31.2%); fix drops by 3.9 points while explain increases by 2.8 points.

The average degradation across both model sizes is approximately 1%, consistent with the paper's claim of "~1% degradation for the pass@1 metric on 3B and 8B models respectively." However, this average masks task-specific variance — the 8B model's fix performance degrades by 3.9 points while its explain performance improves, suggesting the context extension may differentially affect different reasoning modes.

For instruct models, the long-context versions actually show slight improvements:

  • Granite-3b-Code-Instruct-2K: synthesis 39.6%, fix 27.3%, explain 26.0%, average 31.0%
  • Granite-3b-Code-Instruct-128K: synthesis 41.4%, fix 26.2%, explain 25.1%, average 30.9%
    • Essentially flat (31.0% → 30.9%); synthesis improves by 1.8, fix and explain drop slightly.
  • Granite-8b-Code-Instruct-4K: synthesis 49.6%, fix 40.9%, explain 40.4%, average 43.6%
  • Granite-8b-Code-Instruct-128K: synthesis 51.4%, fix 38.3%, explain 38.9%, average 42.9%
    • Average drops by 0.7 points (43.6% → 42.9%); synthesis improves by 1.8, fix and explain drop by 2.6 and 1.5 respectively.

The paper attributes the instruct model improvements to "our new long-context synthetic data added to instruction tuning." This is a key finding: the synthetic data, designed for long-context tasks, appears to provide positive transfer to short-context code synthesis even though it was not designed for that purpose. The fix and explain tasks see slight degradation, likely because the synthetic data does not emphasize bug-fixing or code explanation as strongly as the original instruction tuning mixture.

Figure 3 provides a focused view of HumanEval (Python only) performance, comparing short and long-context models. The base models show "a slight degradation" while the instruct models show "an improvement with long-context scaling." The figure visually reinforces the paper's claim that long-context extension does not come at a meaningful cost to short-context Python code generation.

Ablation Studies and Robustness Checks

Per-language length upsampling: The paper describes downsampling documents under 4096 tokens to 10% retention and oversampling longer sequences on a per-language basis (Section 2.1), stating this was "critical for long-context continual pretraining." However, no ablation study comparing with and without this upsampling is reported. The claim that it is "critical" appears to be based on internal experimentation not presented in the paper. This is a significant omission — the upsampling procedure (10% retention rate, per-language balancing) represents a specific design choice, and without an ablation, the reader cannot assess how sensitive the results are to these parameters or whether the repository-level packing alone would suffice.

Progressive vs. one-shot context extension: The paper uses a progressive doubling schedule (8K → 16K → 32K → 64K → 128K) with per-stage RoPE theta optimization. No comparison to a one-shot extension directly to 128K is provided, nor is there an ablation showing whether the progressive approach outperforms a single-stage jump with the optimal RoPE theta. Prior work (Xiong et al., 2023) demonstrated benefits of progressive schedules, but the paper relies on that precedent rather than providing its own evidence that the progressive approach was necessary for these specific models and data.

RoPE theta values per stage: The paper reports the specific RoPE theta values used (100K, 250K, 500K, 2M, 10M) and states they were found through "search for the optimal RoPE theta" at each stage. No ablation comparing alternative theta values or theta scheduling strategies is reported. The reader cannot assess whether the chosen values are near-optimal, whether a different schedule (e.g., exponential interpolation) would perform similarly, or whether the sensitivity to theta is high enough that hyperparameter search is necessary.

Repository-level packing vs. random concatenation: The repository-level file packing with DAG-based ordering is described as a key innovation. No direct ablation comparing DAG-ordered packing to random interleaving of files from the same repository is presented in the paper's results. The claim that the ordering algorithm is "critical for long-context continual pretraining" (Section 2.1) is stated but not empirically validated with visible results. This is one of the paper's most significant missing experiments — it is the central data engineering contribution, and without an ablation, the evidence for its importance is indirect (the model works well after training on DAG-ordered data, but we don't know how much the ordering specifically contributes).

Short vs. long-context instruction data mixture ratio: The instruction tuning uses a combination of short and long-context data, but the mixing ratio is never specified. No ablation varying the ratio is reported, so the reader cannot assess whether the balance is optimal, whether more long-context data would improve RepoQA at the cost of HumanEvalPack degradation, or whether a different ratio would produce a different tradeoff curve.

PRM/ORM choice: This paper does not use process reward models or outcome reward models — the instruction models are fine-tuned with standard supervised fine-tuning, and evaluation uses greedy decoding. This is not an ablation per se but rather a design choice to keep verification simple.

Effect of multi-turn loss masking: The paper describes using a multi-turn loss mask that computes loss only on assistant response tokens (Section 2.2). No ablation comparing this to full-sequence loss (predicting both user and assistant turns) is reported. The paper cites Wang et al. (2023a) as precedent, but does not validate that the masking approach matters for these specific models and data.

EOS token placement during instruction tuning: The paper notes that "we append an EOS token after each response from the model to prevent runaway generation during inference." No ablation testing the effect of EOS placement (after each response vs. only at sequence end) is presented, though this is a well-known technique and the rationale is clear.

Noise multiplier ablation: The instruction tuning uses a noise multiplier of 5 for input embeddings. No ablation comparing different noise multiplier values or the effect of removing this regularization entirely is reported. This is a minor point but means the reader cannot assess whether this hyperparameter matters or is an incidental carryover from prior work.

Impact of training data languages on evaluation languages: The continual pretraining focuses on Python, C, C++, Go, Java, JavaScript, and TypeScript. Rust is evaluated (in RepoQA and HumanEvalPack) but is not listed as a training language. The paper does not discuss or ablate the effect of this language gap, though the Rust results are reported alongside the other languages. The fact that the 8B 128K model achieves 74.0% retrieval at threshold 0.0 on Rust (Table 3) — outperforming several explicitly trained languages — suggests the model generalizes across languages or that the Rust repositories in RepoQA are simpler than those for other languages. Without an ablation controlling for language presence in training data, this cannot be resolved.

Negative result — short-context models fail catastrophically beyond training length: This is not an ablation but a consistently replicated negative result that validates the evaluation methodology. Across LCC (Tables 1), RepoBench-P (Table 2), RepoQA (Tables 3, 4), and Key Retrieval (Figure 2), the short-context models' performance collapses when sequences exceed their training context length. This serves as a sanity check: the benchmarks genuinely test long-context capability, and the short-context models' near-zero scores confirm that the long-context models' gains are not simply due to the benchmarks being easy or the evaluation being lenient.

Critical Assessment

Claim 1: "Long-context models achieve significant improvements on long-context tasks over short-context counterparts"

What the experiments demonstrate: This claim is overwhelmingly supported for the specific long-context benchmarks tested. Tables 1–4 and Figures 1–2 show massive, consistent improvements across LCC, RepoBench-P, RepoQA, and Key Retrieval. The evidence is particularly strong because the improvements are large in absolute terms (+30–60 percentage points in many cases) and consistent across both model sizes (3B, 8B) and both model types (base, instruct).

What the experiments do not demonstrate: The paper does not show that the improvements scale to context lengths beyond 32K for the code completion benchmarks (LCC and RepoBench-P only go to 32K) or beyond 16K for RepoQA. The model is trained for 128K, but the longest evaluation context is 32K for completion and 16K for retrieval. The Key Retrieval benchmark (Figure 2) does test up to 128K, but the paper describes it as "relatively easy" and shows perfect performance, which provides limited discriminative power. The paper does not test at intermediate lengths between 32K and 128K (e.g., 64K, 96K) for any benchmark, leaving a significant gap in the evaluation coverage. The claim of "up to 128K tokens" support is partially validated by Key Retrieval but not by the more realistic code understanding benchmarks.

Missing experiments: The paper would be strengthened by evaluating LCC and RepoBench-P at 64K and 128K context lengths to demonstrate that the model's code completion capability actually extends to the claimed maximum. The current evaluation at only 4K–32K for these benchmarks leaves open the possibility that the model's effective context length is closer to 32K–64K for code completion tasks, with the full 128K only useful for simpler retrieval tasks.

Claim 2: "Long-context extension does not cause noticeable performance degradation on regular code completion benchmarks"

What the experiments demonstrate: Table 5 and Figure 3 provide reasonable evidence for this claim. The average degradation on HumanEvalPack is approximately 1% for both base model sizes, and the instruct models actually show slight improvements in synthesis (offsetting small drops in fix and explain). The paper is appropriately measured in its language: "does not significantly change" (Section 3.4) rather than claiming zero degradation, and the ~1% figure is stated explicitly.

Nuances and caveats: The "~1%" figure is an average across tasks and languages that masks variance. The 8B base model drops 3.9 points on the fix task — a 13% relative decline — which is arguably "noticeable" for users who primarily use the model for bug-fixing. The synthesis task, which is the most commonly cited HumanEval metric, shows consistent small improvements for instruct models and small degradations for base models. The paper's claim holds on average but users should be aware of task-specific variance.

Missing experiments: The paper evaluates only HumanEvalPack for short-context preservation. Additional short-context benchmarks (MBPP, MultiPL-E, code translation tasks, code explanation benchmarks) would strengthen confidence that the preservation is broad rather than specific to HumanEvalPack. The paper also does not evaluate whether the slight synthesis improvement for instruct models (e.g., 8B: 49.6% → 51.4%) is statistically significant or within noise — again, no confidence intervals are provided.

Claim 3: "Context extension requires only lightweight continual pretraining (0.1% of original data)"

What the experiments demonstrate: The paper reports that the final models were trained on "an extra 4B tokens which is only 0.1% of original pretraining data" (Section 2.1). All results in Tables 1–5 come from models trained with this 4B-token budget, so the claim is internally consistent — the reported performance was achieved with this budget.

What the experiments do not demonstrate: The paper does not show that 4B tokens is necessary or optimal. No ablation varies the training budget — we don't know whether 2B tokens would achieve similar results, or whether 8B tokens would further improve performance. The 0.1% figure is descriptive (this is what was used) but the paper implies it is noteworthy by comparing it to the original pretraining data volume. Without budget ablation, we cannot assess whether the approach is efficient relative to a lower-budget baseline or whether there are diminishing returns beyond some threshold.

Missing experiments: Training budget ablation (e.g., 1B, 2B, 4B, 8B additional tokens) would characterize the data efficiency of context extension and help practitioners decide how much continual pretraining to invest. Additionally, the paper does not report the original pretraining data volume for the Granite models, so the "4B is 0.1%" claim relies on the reader trusting the arithmetic without seeing both numbers. The original Granite code model paper (Mishra et al., 2024) should contain this figure, but it is not stated in the current paper for verification.

Claim 4: "Repository-level file packing with dependency-aware ordering enables long-context capability"

What the experiments demonstrate: The models trained with this data preparation procedure achieve strong long-context performance, which is consistent with the claim. The paper describes the packing methodology in detail and states it was "critical for long-context continual pretraining" (Section 2.1).

What the experiments do not demonstrate: There is no ablation comparing DAG-ordered packing to alternative packing strategies (random file order, alphabetical order, file-size order, random interleaving of files from different repositories). Without this comparison, the claim that the specific dependency-aware ordering is "critical" is an assertion backed by the model's overall performance but not by a controlled experiment. It is possible that any packing strategy that keeps files from the same repository together would achieve similar results, with the dependency ordering contributing only marginal gains. It is also possible that the ordering matters less than the simple fact of having long, coherent (same-repository) sequences as opposed to the short, independent sequences of the original pretraining.

Missing experiments: An ablation comparing (a) DAG-ordered repository packing, (b) random file order within repositories, and (c) naive cross-repository concatenation would directly test the contribution of the dependency-aware ordering. Until such an experiment is conducted, the paper's strongest contribution — the dependency-aware packing algorithm — remains empirically unvalidated as a causal contributor to performance.

Claim 5: "Bootstrapped synthetic instruction data enables long-context instruction following"

What the experiments demonstrate: The instruct models (Tables 4, Figures 1–2) achieve strong long-context instruction following after fine-tuning on the mixture that includes the bootstrapped synthetic data. In particular, RepoQA scores for instruct models are generally higher than for base models (e.g., 8B instruct at threshold 0.5: 61.6% vs. 8B base: 46.6%), and the instruct models close the gap to exact match more effectively. This is consistent with the synthetic data contributing positively.

What the experiments do not demonstrate: The paper does not show how much of the instruct model's long-context capability comes from the synthetic data versus the short-context instruction data mixture. The instruct models are trained on a combination of both, and the short-context data includes diverse code-related tasks (CommitPackFT, API calling, multi-turn interactions) that may themselves contribute to the model's ability to follow instructions in long contexts. There is no ablation comparing instruction tuning with only short-context data (to measure how much long-context capability transfers from the base model) versus the full mixture including synthetic data.

Missing experiments: An ablation comparing (a) long-context base model without instruction tuning, (b) long-context base model + short-context instruction data only, and (c) long-context base model + full mixture (short + synthetic long) on RepoQA would quantify the specific contribution of the synthetic data. The paper's claim that the improvements are "attributed to the knowledge learned from newly introduced synthetic long data for instruction tuning" (Section 3.3) is plausible but not isolated experimentally.

Genuine Weaknesses in the Experimental Design

Narrow context-length evaluation range for realistic benchmarks. The paper's title and abstract emphasize "128K context," but the most realistic code understanding benchmarks — LCC and RepoBench-P — are evaluated only up to 32K tokens. RepoQA uses a fixed 16K context. The only benchmark that tests the full 128K range is the synthetic Key Retrieval task, which the authors themselves describe as "relatively easy." This creates a significant mismatch between the claimed capability (128K) and the demonstrated capability on realistic code tasks (demonstrated to 16K–32K). The paper does not explain why 64K or 128K evaluations on LCC/RepoBench-P were not conducted, though the likely reason is that these benchmarks were not originally designed for such long contexts and may not have sufficient data at those lengths even after rebalancing.

No statistical confidence reported. All tables present point estimates without confidence intervals, standard deviations, or statistical tests. With RepoQA's 500 subtasks (100 per language), LCC's ~100 samples per bucket, and HumanEvalPack's 6 languages × 3 tasks, the sample sizes are modest enough that sampling variance could meaningfully affect the comparisons. The paper would be strengthened by reporting bootstrap confidence intervals or at minimum standard deviations for the per-bucket averages.

Single model family evaluation. All results are on Granite 3B/8B models. The paper does not demonstrate that the context-extension recipe transfers to other model architectures, other model sizes, or other code model families. This is particularly relevant because the Granite models use full attention with RoPE — models using different positional encodings (ALiBi, NoPE) or different attention mechanisms (grouped-query attention, multi-query attention) may respond differently to the RoPE-theta adjustment and progressive training schedule.

No comparison to other long-context code models. The paper compares long-context Granite models only to short-context Granite models. It does not benchmark against other long-context open-source code models (Code Llama with extended context, DeepSeek Coder, StarCoder2 with longer context support) or against proprietary models. This makes it impossible to assess whether the achieved long-context performance is competitive or merely improved relative to a very low baseline. The RepoQA benchmark was specifically designed for cross-model comparison (Liu et al., 2024), but the paper does not place its results in the context of the broader RepoQA leaderboard.

Language coverage mismatch between training and evaluation. The paper trains on Python, C, C++, Go, Java, JavaScript, and TypeScript but evaluates on Rust (RepoQA, HumanEvalPack) and C# (LCC). The Rust results are surprisingly strong (8B instruct at threshold 1.0: 40.0%, second only to Python's 48.0%), while C# is only tested within LCC where per-language breakdowns are not provided. The paper does not discuss whether the Rust performance reflects genuine cross-language generalization or artifacts of the specific evaluation data.

Repository-level packing evaluation is indirect. The paper's most novel data engineering contribution — DAG-based dependency ordering — is never directly evaluated. We don't know whether models trained with this ordering actually learn to attend across files along dependency chains, or whether they simply learn to handle longer sequences and the specific ordering is incidental. An analysis of attention patterns on RepoQA examples (e.g., does the model attend to the file that defines an imported function?) would provide direct evidence for the mechanism the paper claims is at work.

Missing Experiments That Would Have Strengthened the Paper

  1. LCC and RepoBench-P evaluation at 64K and 128K to validate the full claimed context length on realistic code completion tasks.
  2. Ablation of packing strategy (DAG-ordered vs. random-same-repo vs. cross-repo concatenation) to quantify the contribution of dependency-aware ordering.
  3. Ablation of instruction data mixture (short-context only vs. full mixture) to isolate the synthetic data's contribution.
  4. Training budget ablation (1B, 2B, 4B, 8B additional tokens) to characterize data efficiency.
  5. Comparison to at least one external long-context code model on RepoQA to contextualize the absolute performance.
  6. Attention pattern analysis on RepoQA examples to verify that the model uses cross-file attention along dependency chains.
  7. Confidence intervals or standard deviations on all reported metrics to enable assessment of whether observed differences are statistically meaningful.
  8. Evaluation on additional short-context benchmarks (MBPP, MultiPL-E) to broaden the short-context preservation evidence.
  9. Perplexity or loss measurements at different context lengths to complement the task-based evaluations with a more continuous measure of the model's effective context utilization.

6. Limitations and Trade-offs

The Claimed 128K Context Length Is Validated Only on a Synthetic Retrieval Task, Not on Realistic Code Understanding Benchmarks

The assumption or constraint. The paper's title, abstract, and central claim assert that the models support "effective context windows of up to 128K tokens." However, the evaluation of realistic code understanding tasks — the ones that matter for the repository-level coding and software development agents that motivate the work — is conducted at much shorter context lengths. Section 3.1 describes the benchmarks: LCC is evaluated at context lengths up to 32K (Table 1), RepoBench-P up to 32K (Table 2), and RepoQA uses a fixed 16K context (Tables 3, 4). The only benchmark that tests the full 128K range is Key Retrieval (Figure 2), which the paper itself characterizes as "relatively easy for long-context code LLMs." The paper does not explicitly acknowledge this gap between claimed and evaluated context length on realistic tasks — there is no statement in the text that the code completion or code understanding evaluations stop at 16K–32K while the model is advertised at 128K.

The consequence. A practitioner deploying this model for repository-scale code completion at 128K context has no direct evidence that the model's code generation quality holds up at those lengths. The Key Retrieval result (Figure 2) demonstrates that the model can attend to specific tokens at position 128K — a necessary condition for long-context utility — but it does not demonstrate that the model can reason about code at that scale, maintain syntactic and semantic coherence across 100K+ tokens of context, or effectively integrate information from distant parts of a large repository for code completion. The LCC and RepoBench-P results show that performance is relatively flat from 4K to 32K for the long-context models (e.g., 8B base on RepoBench-P: 42.7%, 44.0%, 44.8%, 44.5% at 4K, 8K, 16K, 32K), which is encouraging but does not guarantee flat performance at 64K or 128K — attention quality could degrade, the model could lose track of early-context information, or generation quality could decline for reasons not captured by the synthetic retrieval task.

This limitation is particularly consequential because the paper's motivation is explicitly about repository-level coding and software agents (Section 1): "With the emergence and development of repository-level coding tasks and software development agents, long context length becomes an important feature." Real repositories routinely exceed 32K tokens — a medium-sized Python project with a few dozen files can easily span 50K–100K tokens. If the model's effective code understanding degrades between 32K and 128K, a significant class of the motivating use cases is not actually served.

What evidence exists in the paper. The evidence gap is visible in the evaluation tables themselves. Tables 1 and 2 report LCC and RepoBench-P only to 32K. Tables 3 and 4 report RepoQA only at 16K. Figure 2 is the sole 128K evaluation, and the paper's own commentary on it — "we tend to view that this level of retrieval is relatively easy" — acknowledges its limited discriminative power. The paper does not provide per-length perplexity curves, attention entropy measurements, or any continuous metric of context utilization across the 4K–128K range that would allow a reader to extrapolate task performance.

Mitigation status. The paper does not attempt to address this limitation. There is no discussion of why 64K or 128K evaluations on LCC or RepoBench-P were not conducted, no extrapolation argument from the 32K results to longer contexts, and no acknowledgment that the evaluation coverage is incomplete relative to the claims. The Key Retrieval benchmark provides a partial signal — perfect retrieval at 128K confirms that the attention mechanism physically functions at that length — but the paper does not argue that this is sufficient evidence for the code understanding claims, nor does it flag the gap as an area for future work. A practitioner must treat the "128K" claim as validated for simple information lookup but unvalidated for the generative code tasks that constitute the primary use case.

The Evaluation Does Not Compare Against Any External Long-Context Code Model, Making Absolute Performance Uncontextualized

The assumption or constraint. The paper's entire evaluation framework compares the 128K Granite models exclusively against their short-context predecessors. Section 3 describes comparisons against "original Granite code models" and "their short-context counterparts." There are no head-to-head benchmarks against other long-context open-source code models (Code Llama with extended context, DeepSeek Coder variants with long-context support, StarCoder2 long-context versions, or CodeGemma models) or against proprietary models (GPT-4, Claude, Gemini). The evaluation design answers the question "does context extension improve over the short-context Granite baseline?" but does not answer "are the resulting models competitive with other available long-context code models?"

The consequence. A practitioner choosing between available long-context code models cannot use this paper to make an informed decision. The RepoQA benchmark was specifically designed as a cross-model evaluation framework (Liu et al., 2024), and results for other models on this benchmark exist in the literature, but this paper does not place its numbers in that context. The 8B instruct model's 61.6% retrieval accuracy at threshold 0.5 on RepoQA could be excellent (if other open-source models score 40–50%) or mediocre (if competing models score 70–80%), and the paper provides no way to distinguish these scenarios. Similarly, the HumanEvalPack pass@1 scores for the 128K instruct models (e.g., 8B: 51.4% synthesis) are reported without comparison to other 8B-class code models with long-context support, leaving unclear whether the long-context extension preserves competitive short-context performance or merely preserves it relative to an already-modest baseline.

The paper's framing in Section 1 — "While many proprietary large language models... support very long context windows, most open-source code language models could only provide relatively short context windows" — sets up an expectation that the paper is addressing the open-source vs. proprietary gap. But without any comparison to other open-source long-context models (which do exist, though perhaps not with the same training transparency) or to proprietary baselines, the reader cannot assess how much of that gap has been closed.

What evidence exists in the paper. The absence of external comparisons is visible in every results table. Tables 1–5 contain only Granite model variants (short-context base, long-context base, short-context instruct, long-context instruct) with no rows for competing models. Figure 1 compares short and long-context Granite instruct models on RepoQA but places no external baselines on the chart. The paper's discussion of results (Section 3.2–3.4) frames all findings as relative improvements over short-context Granite versions rather than as competitive positioning.

Mitigation status. The paper does not attempt to address this limitation or acknowledge it as a gap. There is no statement that external comparisons were beyond scope, no reference to existing RepoQA leaderboards that readers could consult for context, and no suggestion that such comparisons would be valuable future work. The choice to limit evaluation to within-model-family comparisons was clearly deliberate (the paper is about extending Granite, not about competing with other model families), but the consequence is that the reported numbers float in isolation without the external anchoring that would make them actionable.

The Core Data Engineering Innovation — Dependency-Aware Repository Packing — Is Not Ablated, Leaving Its Causal Contribution Unknown

The assumption or constraint. The paper's most distinctive technical contribution is the repository-level file packing algorithm with DAG-based dependency ordering (Section 2.1). The paper describes this methodology in detail — building a directed acyclic graph from file imports, breaking cycles, performing topological sorting, arranging non-connected files by depth-first folder traversal — and states it was "critical for long-context continual pretraining." However, the paper provides no experimental evidence that this specific ordering algorithm contributes to model performance beyond what simpler packing strategies would achieve. There is no ablation comparing (a) DAG-ordered repository packing against (b) random ordering of files within the same repository, (c) alphabetical or file-size ordering, or (d) naive concatenation of files from different repositories to the same sequence length.

The consequence. The paper's most novel methodological contribution is empirically unvalidated as a causal factor in the reported performance. It is possible — and the paper provides no evidence to distinguish among these possibilities — that:

  • The DAG ordering is indeed critical, and performance would collapse without it.
  • Any strategy that keeps files from the same repository together (regardless of ordering) would perform similarly, because the key factor is simply coherence (all files relate to the same project) rather than dependency-specific ordering.
  • Simply having long sequences at all (regardless of file-level coherence) is the dominant factor, and the packing strategy is incidental — the model learns long-range attention from any long sequences, and the repository structure is a nice-to-have rather than a requirement.

Without an ablation, a practitioner reimplementing this approach cannot know which aspects of the data preparation pipeline are load-bearing and which are incidental. If the dependency graph construction and topological sorting are difficult to implement (requiring language-specific import parsing, cycle-breaking heuristics, and per-repository graph analysis), but random same-repo packing works nearly as well, the implementation complexity is wasted. Conversely, if the DAG ordering is critical, the paper's failure to demonstrate this means its most important practical recommendation is undersupported.

This limitation is particularly significant because the data preparation methodology is the paper's primary contribution to the how of context extension. The RoPE theta adjustment and progressive training schedule are adapted from prior work (Xiong et al., 2023; Fu et al., 2024); the instruction tuning data mixture follows established practice. The dependency-aware packing is where the paper claims novelty, but that novelty is asserted rather than demonstrated.

What evidence exists in the paper. The paper states the packing approach was "critical" (Section 2.1) but provides no ablation evidence. All results in Tables 1–5 come from models trained with the full DAG-ordered packing pipeline, with no alternative packing strategies evaluated. The paper does not report training dynamics (loss curves, gradient statistics, attention pattern analyses) that might show whether the model trained on DAG-ordered data learns different cross-file attention patterns than a model trained on randomly ordered data. There is no analysis of whether model attention on RepoQA examples actually follows import dependency chains (i.e., when retrieving a function, does the model attend to the file that defines the imported dependency?).

Mitigation status. The paper does not acknowledge this as a limitation or call for future ablation work. The "critical" claim is presented as an established finding rather than as an assertion requiring validation. The ablation study section of this analysis (Section 5) notes this gap, but the paper itself does not address it. For a methodology paper where the data engineering pipeline is the central contribution, the absence of even a minimal ablation (e.g., DAG-ordered vs. random-same-repo on a subset of the training) is a significant evidentiary weakness.

Training Budget Efficiency Claims ("Only 0.1% of Original Pretraining Data") Are Not Contextualized by Original Pretraining Data Volume or Budget Ablations

The assumption or constraint. The paper emphasizes that the long-context extension required "only 0.1% of original pretraining data" (Section 2.1) — 4 billion additional tokens. This figure is presented as evidence of the approach's efficiency and is framed as a key practical advantage: context extension is cheap relative to pretraining from scratch. However, the paper does not report the original pretraining data volume for the Granite models, does not provide an ablation varying the continual pretraining budget (e.g., 1B, 2B, 4B, 8B additional tokens), and does not show learning curves for long-context performance as a function of additional tokens.

The consequence. A practitioner cannot assess whether the 4B token budget is near-optimal, wasteful, or insufficient. Several scenarios are possible, and the paper provides no evidence to discriminate among them:

  • Diminishing returns set in early: 2B tokens might achieve 90% of the reported performance, making the 4B budget twice as expensive as necessary for similar results.
  • Returns are still linear: 8B tokens might yield substantially better performance, meaning the paper's results are not at the ceiling and additional investment would be worthwhile.
  • The 0.1% figure is misleading: If the original Granite models were trained on anomalously large data volumes (e.g., 4T tokens rather than a more typical ~1T for models of this size), then "0.1% = 4B" is an artifact of a large denominator rather than evidence that context extension is inherently cheap. A model family pretrained on 400B tokens would need only 400M tokens to achieve the same "0.1%" ratio — does the approach still work at that scale?

More fundamentally, without knowing the original pretraining data volume, the "0.1%" claim is unverifiable. The paper states the Granite code models are from Mishra et al. (2024), and readers can consult that reference for pretraining details, but the self-contained paper does not provide the information needed to evaluate its own efficiency claim.

The 500 steps per stage with batch size 32 at progressively doubling context lengths implies a specific token budget that is reported (4B total), but the relationship between this budget and the achieved performance is unexplored. If a practitioner wanted to extend a model with a different pretraining data volume, should they target 0.1% of that volume? 4B tokens regardless? Some other heuristic? The paper offers no guidance.

What evidence exists in the paper. The paper states the 4B token figure and the 0.1% comparison in Section 2.1: "The final models are trained for an extra 4B tokens which is only 0.1% of original pretraining data." The 500 steps × batch size 32 × progressive context lengths yields this 4B figure, but the original pretraining token count is not stated anywhere in the paper. No training budget ablation is presented; no learning curves showing long-context task performance vs. additional training tokens are shown; no analysis of whether performance saturates at 4B tokens or continues to improve is provided.

Mitigation status. The paper does not treat this as a limitation requiring mitigation or future work — it presents the 0.1% figure as an established finding rather than as a preliminary result needing validation. The absence of budget ablation means the efficiency claim is descriptive (the reported performance was achieved at this budget) but not prescriptive (it does not tell practitioners what budget they should target for their own context extensions). The paper does not discuss the relationship between pretraining data volume and the required continual pretraining budget for context extension, leaving an important practical question unanswered.

The Long-Context Instruct Model's Performance Depends on Synthetic Data Whose Quality Is Constrained by the Short-Context Generator Model

The assumption or constraint. The synthetic long-context instruction data is generated using the original Granite-8b-Code-Instruct-4K model (Section 2.2), which has a 4K context window and therefore cannot see the full repository-level packed documents at once. The paper describes generating multi-turn instruction-response pairs by parsing the packed document into constituent functions and methods, then using the 4K model to generate responses — but the 4K model's limited context window means it cannot ground its responses in cross-file dependencies that span beyond 4K tokens. The training data is explicitly constructed to teach the model to reason across long contexts, yet the data generation process itself is constrained by a model that cannot do long-context reasoning.

The paper acknowledges this choice as a deliberate one — "we generate... with our original Granite-8B-Code-Instruct model to avoid the dependency on an existing long context model" — but does not discuss the quality implications of using a context-limited generator to create training data for long-context tasks.

The consequence. The synthetic data may be of limited quality for tasks that genuinely require cross-file reasoning beyond 4K tokens. For the retrieval task (Task 1 in Section 3.4), the ground-truth function implementation can be extracted programmatically without model generation, so quality is not an issue. But for the explanation task (Task 2: "asks for an explanation of that implementation using available documentation") and the implementation-from-context task (Task 3: "generates instructions for implementing the sampled functions based on the remaining documentation and code with the function excluded"), the short-context generator model may produce responses that are locally correct but miss dependencies on code or documentation more than 4K tokens away from the target function.

This could create a subtle quality ceiling on the instruct model's long-context capability: it learns to follow instructions over long contexts, but the training signal for genuinely cross-file reasoning is only as good as a 4K-context model can provide. The strong RepoQA results for the instruct model (Table 4: 8B instruct at threshold 0.5 scores 61.6% vs. 8B base at 46.6%) suggest the synthetic data does improve retrieval — but RepoQA tasks require finding a function based on a description, which is closer to the retrieval task (where ground-truth extraction is used) than to the explanation or implementation tasks. The paper does not evaluate on benchmarks that specifically test the explanation and implementation skills that the synthetic data was designed to teach, so the quality of the generated training signal for those tasks is unvalidated.

More fundamentally, the approach introduces a bootstrapping paradox: to train a model to reason across 128K contexts, you use a model limited to 4K contexts to generate training examples. The training examples can only capture cross-file reasoning to the extent that (a) the relevant cross-file information happens to fit within 4K tokens, or (b) the task decomposition (one function at a time) does not require cross-file reasoning. If genuinely long-range dependencies are important for repository understanding — and the paper's entire motivation argues they are — then the synthetic data generation process may systematically underrepresent them.

What evidence exists in the paper. The paper does not provide any direct quality assessment of the synthetic data. There is no evaluation of how often the 4K generator model produces incorrect or incomplete responses due to missing cross-file context, no comparison of synthetic data quality vs. what a long-context model would produce, and no ablation comparing instruct models trained with short-context-generated synthetic data vs. alternative data sources. The paper relies on the downstream RepoQA and HumanEvalPack results as indirect evidence that the synthetic data is useful, but does not isolate the contribution of synthetic data quality from the contribution of simply having more multi-turn instruction examples.

The RepoQA results for instruct models are consistently better than base models (e.g., 8B: instruct 61.6% vs. base 46.6% at threshold 0.5), which is consistent with the synthetic data helping, but this does not tell us whether the data is near-optimal or significantly degraded by the generator model's context limitation.

Mitigation status. The paper acknowledges the bootstrapping constraint — "to avoid the dependency on an existing long context model" — but frames it as a feature (independence from proprietary models) rather than as a limitation of data quality. There is no discussion of whether the short-context generator's limitations create a quality ceiling for the instruct model, no comparison to what a long-context generator would produce, and no suggestion that improving the synthetic data generation pipeline (e.g., by using the newly created long-context base model as the generator in a second iteration) would be valuable future work. The approach successfully avoids the dependency problem, but the paper does not address the quality tradeoff this choice entails.

Short-Context Performance Preservation Is Evaluated on Only One Benchmark Family (HumanEvalPack), with Task-Specific Variance Masked by Averaging

The assumption or constraint. The paper's claim that long-context extension does not cause "noticeable performance degradation on regular code completion benchmarks" (abstract) and "does not significantly change the short-context generic capability" (Section 3.4) is supported by a single evaluation: HumanEvalPack (Table 5), which tests synthesis, fixing, and explanation across 6 programming languages. The paper does not evaluate on other widely-used short-context code benchmarks such as MBPP, MultiPL-E, DS-1000, CodeContests, or any benchmark testing code translation, code summarization, or documentation generation. The "~1% degradation" figure is an average across three tasks and six languages in HumanEvalPack, which masks task-level and language-level variance.

The consequence. A practitioner deploying the long-context model for short-context tasks cannot be confident that the preservation holds beyond the specific task distribution of HumanEvalPack. The paper's own data reveals task-specific variance that complicates the "no noticeable degradation" narrative:

  • Granite-8b-Code-Base-128K drops from 29.1% to 25.2% on the fix task — a 3.9 percentage point decline (13% relative). For a practitioner using the model primarily for bug-fixing, this is arguably "noticeable."
  • Granite-8b-Code-Instruct-128K drops from 40.9% to 38.3% on fix and from 40.4% to 38.9% on explain — consistent small declines across both non-synthesis tasks.
  • The synthesis task, which is the most commonly cited HumanEval metric and the one the paper highlights in Figure 3, actually improves for instruct models (8B: 49.6% → 51.4%), which partially offsets the other declines in the average.

The average degradation may be ~1%, but the task-level results suggest a pattern: long-context extension may slightly degrade code understanding and repair capabilities while preserving or improving code generation from specifications. Without evaluation on additional short-context benchmarks that test different aspects of code capability, it is unclear whether this pattern generalizes or is specific to HumanEvalPack's particular task formulations.

Additionally, the paper does not evaluate whether the long-context models show any degradation in instruction-following fidelity on short contexts — do they become more likely to ignore instructions, produce longer-than-necessary responses, or hallucinate repository context when given only a single file? These are practical failure modes that HumanEvalPack's pass@1 metric (which only checks functional correctness of the generated code) does not capture.

What evidence exists in the paper. Table 5 provides the per-task breakdown that reveals the variance. Section 3.4 states the average degradation (~1%) and notes that "interestingly, we notice improvements in HumanEval performance of long-context instruct models, which we attribute to our new long-context synthetic data added to instruction tuning." Figure 3 focuses on HumanEval (Python only) and visually shows the pattern of slight base model degradation and instruct model improvement. The paper does not provide per-language breakdowns for HumanEvalPack that would reveal whether preservation is language-dependent (e.g., does Python preservation differ from Rust preservation?).

The absence of additional short-context benchmarks is visible in the evaluation section (Section 3.1), which lists HumanEvalPack as the sole short-context evaluation. The paper does not discuss why MBPP, MultiPL-E, or other standard code benchmarks were not included, nor does it acknowledge the narrow scope of short-context evaluation as a limitation.

Mitigation status. The paper does not treat the narrow short-context evaluation scope as a limitation. The claim of "no noticeable degradation" is presented as an established finding without caveats about benchmark coverage. The paper does not call for broader short-context evaluation in future work, does not discuss the task-specific variance in the HumanEvalPack results (the fix-task degradation is never mentioned in the text), and does not provide the per-language breakdowns that would allow readers to assess whether preservation is uniform across the programming languages that matter for their use case. The paper's conclusion (Section 4) summarizes: "without significantly affecting the short-context generic capability" — a claim that the single-benchmark evaluation supports on average but not with the breadth or granularity that would constitute strong evidence of "generic capability" preservation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new architecture, a new training objective, or a new theoretical framework for long-context modeling. It is an engineering contribution—a specific, validated recipe for extending short-context code models to 128K tokens—but one whose implications shift the practical calculus for open-source code model development in three specific ways.

First, it reframes context extension from an architecture problem to a data engineering problem. The dominant precedent for this claim is Xiong et al. (2023), which established that RoPE base frequency adjustment can physically enable longer-range attention. But prior work placed the emphasis on the RoPE adjustment itself—finding the right theta, designing the right interpolation schedule—with data treated as a secondary consideration (just make sequences longer). This paper inverts that emphasis. The central finding is that with full attention preserved exactly (no sparse approximations, no architectural modifications), and with only 4B additional training tokens (0.1% of the original pretraining budget, per Section 2.1), a model can learn to use 128K context effectively if the training data is structured to teach cross-file dependency reasoning. The RoPE adjustment is necessary but not sufficient; the data engineering is the active ingredient.

This reframing has concrete consequences for how research teams should allocate effort. If the key to context extension is data structure (how files are ordered, whether dependencies are respected, whether long sequences are adequately represented per language) rather than positional encoding formulas, then improvements in data preparation methodology—better dependency graph construction, more sophisticated repository structure analysis, language-specific ordering heuristics—are higher-leverage investments than further RoPE interpolation research. Teams that previously spent effort tuning positional encoding schedules should redirect toward program-analysis-based data engineering.

Second, it demonstrates that full-attention preservation is viable at 128K scale for code models in the 3B–8B parameter range. The paper explicitly rejects sparse or linear attention (Section 2.1), using Flash Attention 2 and Ring Attention to handle the computational cost. This matters because it eliminates an entire category of train-inference mismatch: the model learns attention patterns under the exact same mathematical operation it will use at deployment. The practical implication is that for models in this size range, the "long context vs. short-context quality" tradeoff that sparse attention methods often introduce is not a necessary compromise. Engineering investment in parallelism (Flash Attention 2, Ring Attention, sequence parallelism) can substitute for architectural compromise, and the result is a model that preserves short-context performance to within approximately 1% (Table 5) while adding 128K capability.

This is not a paradigm shift—full-attention long-context training has been done before—but it is a validated existence proof at a specific model scale and domain (code) that provides a concrete reference point for practitioners. A team wondering whether they need to adopt sparse attention for their 7B code model now has evidence that full attention works, with specific parallelism strategies named and specific training budgets quantified.

Third, it provides a reconciliation for the apparent tension between long-context extension and short-context preservation. The paper's HumanEvalPack results (Table 5) show that base models lose approximately 1% on average after context extension, while instruct models actually improve on synthesis tasks. The authors attribute the instruct model improvement to the synthetic long-context instruction data (Section 3.4): "interestingly, we notice improvements in HumanEval performance of long-context instruct models, which we attribute to our new long-context synthetic data added to instruction tuning." This suggests a mechanism where diverse fine-tuning data can not only prevent forgetting during context extension but actively improve short-context capabilities—the multi-turn, repository-grounded synthetic data appears to transfer positively to single-file code generation. This partially resolves the fear that context extension is a zero-sum tradeoff, though the task-specific variance (the 8B instruct model's fix task drops from 40.9% to 38.3%) indicates the preservation is uneven and task-dependent.

Research directions that become more attractive:

  • Data engineering for domain-specific long-context extension. The paper's dependency-aware repository packing is an instance of a more general principle: structure training sequences so that long-range attention learns task-relevant patterns. For legal documents (citation-ordered packing), scientific literature (reference-chain ordering), or multi-document summarization (topical clustering), analogous data engineering strategies become promising.
  • Full-attention long-context training for other modalities and scales. The paper demonstrates viability at 3B–8B for code; extending to larger models (20B–70B) and other domains (legal, biomedical, multilingual) would establish whether the "RoPE adjustment + data engineering + parallelism" recipe generalizes.
  • Synthetic long-context instruction data bootstrapping. The paper's approach of generating long-context instruction data from a short-context model and structured pretraining documents (Section 2.2) is a transferable methodology that other open-source teams can adopt immediately, particularly for domains where long-context instruction datasets are scarce.

Research directions that become less attractive:

  • Sparse attention for code models in the sub-10B range. If full attention with Flash Attention 2 and Ring Attention works at 128K for 8B models with negligible short-context degradation, the case for introducing architectural sparsity—with its attendant train-inference mismatch and potential quality loss—weakens.
  • One-shot context jumps without progressive training. The paper's progressive doubling schedule (8K → 16K → 32K → 64K → 128K) with per-stage RoPE theta search (Section 2.1) aligns with prior findings (Xiong et al., 2023) that progressive extension outperforms single jumps. Researchers proposing one-shot extension methods now face a stronger empirical baseline to beat.

Follow-Up Research This Work Enables

Ablation of dependency-aware ordering against simpler packing strategies to establish whether the DAG-based algorithm actually matters. The paper's most distinctive data engineering contribution—the directed acyclic graph construction from file imports, cycle breaking, topological sorting, and depth-first traversal for unconnected files (Section 2.1)—is stated to be "critical for long-context continual pretraining" but is never empirically validated against alternatives. A controlled experiment training Granite-3B models with identical RoPE schedules, identical training budgets (500 steps per stage, 4B total tokens), and identical per-language upsampling, but comparing three packing strategies—(a) DAG-ordered repository packing, (b) random file ordering within each repository, and (c) cross-repository random concatenation—would directly measure the marginal contribution of dependency-aware ordering. The key metrics would be RepoBench-P EM at 16K and 32K (where cross-file dependencies matter most) and RepoQA retrieval accuracy (where finding a function requires understanding its relationship to the rest of the repository). If DAG ordering provides a large advantage over random same-repo packing, the paper's central contribution gains strong empirical support. If random same-repo packing performs nearly as well, the field learns that repository coherence (all files from the same project) is the active ingredient, not dependency ordering specifically—which simplifies the data preparation recipe and redirects research effort toward repository identification and cohesion rather than import-graph analysis.

Evaluation of LCC and RepoBench-P at 64K and 128K context lengths to validate the claimed context window on realistic code completion benchmarks. The paper's title and abstract assert 128K support, but Tables 1 and 2 evaluate LCC and RepoBench-P only to 32K, and RepoQA only at 16K. The Key Retrieval benchmark (Figure 2) shows perfect retrieval at 128K but is described as "relatively easy." A strong follow-up would rebalance LCC and RepoBench-P for context-length buckets at 64K and 128K (or construct new 64K–128K test sets from the same benchmark distributions if existing data is insufficient), then evaluate the Granite-128K models against the short-context baselines at these lengths. The critical question: does the flat performance curve observed from 4K–32K (e.g., 8B base on RepoBench-P: 42.7%, 44.0%, 44.8%, 44.5%) extend to 64K and 128K, or does performance degrade? If performance holds at 128K, the paper's central claim gains direct evidential support. If performance degrades, the "effective context window" is closer to 32K–64K for code completion tasks, and the 128K claim should be narrowed to retrieval-only tasks. A strong study would also report per-length perplexity on a held-out code corpus from 4K to 128K to provide a continuous measure of context utilization, complementing the task-based benchmarks.

Cross-model-family replication of the RoPE + data engineering recipe to test whether the approach is specific to the Granite architecture. The paper's entire evaluation is on Granite 3B and 8B models, which share a specific architecture (full attention, RoPE, specific pretraining data distribution). A replication applying the same recipe—progressive context doubling with per-stage RoPE theta search, repository-level DAG-ordered packing with per-language upsampling, 4B-token continual pretraining budget—to a different open-source code model family (e.g., DeepSeek Coder 6.7B, StarCoder2 7B, or Code Llama 7B) would establish whether the approach is architecture-agnostic or Granite-specific. The replication should use identical benchmarks (LCC, RepoBench-P, RepoQA, HumanEvalPack) and report both the long-context gains and the short-context preservation relative to the original model. If the recipe transfers, the paper's contribution becomes a general-purpose methodology for the open-source code model community. If it fails on certain architectures (e.g., models using grouped-query attention rather than multi-head attention, or models with different RoPE base frequencies in their original pretraining), the boundary conditions become clearer and the community learns which architectural properties enable lightweight context extension.

Training budget scaling study to characterize the data efficiency of context extension. The paper uses 4B additional tokens (500 steps per stage × batch size 32 × progressive context lengths) and notes this is 0.1% of the original pretraining data, but provides no evidence that 4B is necessary, sufficient, or near-optimal. A budget ablation training Granite-8B models with 1B, 2B, 4B, and 8B additional tokens (by varying training steps per stage) and evaluating on the full benchmark suite would produce a data-efficiency curve for context extension. This curve would answer practical questions: can practitioners achieve 90% of the reported gains with 2B tokens (half the budget)? Do returns diminish sharply after 4B, or does performance continue to improve? The results would also enable comparison to the original pretraining budget—if context extension shows steep diminishing returns at 4B while original pretraining showed linear improvements at that scale, it would strengthen the paper's claim that the model already possesses the necessary attention mechanisms and only needs a modest "nudge" through data exposure. Conversely, if performance continues to improve linearly to 8B+, it would suggest context extension benefits from more substantial investment, potentially changing the economic calculus for teams planning long-context deployments.

Second-iteration synthetic data generation using the long-context model itself to improve instruction data quality. The paper's synthetic instruction data is generated by the original Granite-8b-Code-Instruct-4K model (Section 2.2), which cannot see the full packed document due to its 4K context window. This creates a potential quality ceiling: the training data for genuine cross-file reasoning tasks (explanation, implementation-from-context) is only as good as a 4K-context generator can produce. A follow-up study would use the newly created Granite-8b-Code-Instruct-128K model (or the long-context base model) to regenerate the synthetic instruction data—now with the generator able to see the full repository context—and then fine-tune a fresh instruct model on this higher-quality data. Comparing the resulting model against the current instruct model on RepoQA (particularly at high similarity thresholds where precise retrieval matters) and on a new benchmark specifically testing cross-file explanation and implementation (not currently evaluated) would quantify the quality gain from using a long-context generator. If the improvement is substantial, it establishes a virtuous cycle: each generation of long-context models can produce better training data for the next generation. If the improvement is marginal, it suggests that the task decomposition strategy (one function at a time) already captures most of the relevant context even with a 4K generator, and the bootstrapping limitation is less consequential than it appears.

Attention-pattern analysis on RepoQA examples to verify that dependency-ordered training actually teaches cross-file dependency attention. The paper claims that repository-level file packing with DAG-based ordering teaches the model to attend across files along import dependency chains, but provides no mechanistic evidence. A follow-up study would take trained Granite-128K models, run RepoQA retrieval tasks, and analyze the attention patterns during retrieval: when the model is asked to find a function that uses an imported utility, does it attend to the file that defines that utility? Does the attention pattern follow the topological ordering of the training data (dependencies before dependents)? Comparing attention patterns between models trained on DAG-ordered data and models trained on random-ordered data (if the ablation is conducted) would directly test whether the ordering algorithm shapes the model's internal retrieval strategy or merely provides coherent long sequences. If DAG-ordered training produces attention patterns that mirror import graphs, it validates the paper's central mechanistic claim and suggests that attention analysis can be used to diagnose data preparation quality. If attention patterns are similar regardless of packing order, it suggests the model learns long-range attention in a data-order-agnostic way, and the dependency ordering's benefit (if any) operates through a different mechanism—perhaps improved token-level prediction accuracy due to more predictable file transitions, rather than learned dependency-tracking attention.

Practical Applications and Downstream Use Cases

Repository-scale code completion for open-source IDEs and self-hosted development environments. The 128K Granite code models enable a coding assistant that can ingest an entire small-to-medium repository as context—or a substantial chunk of a larger repository—and provide next-line or next-block code completions that respect project-wide conventions, API usage patterns, and cross-file dependencies. The LCC and RepoBench-P results (Tables 1–2) quantify the benefit: at 16K context, the 8B long-context base model achieves 51.8% EM on LCC and 44.8% on RepoBench-P, compared to 19.1% and 15.7% for the short-context version—absolute improvements of 32.7 and 29.1 percentage points respectively. For a developer working in a self-hosted environment where code privacy prevents using cloud APIs (GPT-4, Claude), deploying the Granite-128K model on local or on-premise hardware provides repository-aware completions that were previously only available from proprietary services. The Apache 2.0 license removes licensing friction for commercial integration into IDE plugins or continuous integration pipelines. The key deployment consideration is hardware: full attention at 128K requires sufficient GPU memory for the KV cache, but the 3B model is small enough to run on consumer GPUs with quantization, making repository-scale code completion accessible on developer laptops rather than requiring datacenter infrastructure.

Needle-in-the-haystack code retrieval for large-scale codebase search and documentation. The RepoQA results (Tables 3–4) demonstrate a capability that directly translates to codebase search tools: given a natural language description of a function, find its exact implementation within a 16K-token repository context. The 8B instruct model achieves 61.6% retrieval accuracy at a 0.5 similarity threshold and 28.2% exact-match retrieval—dramatic improvements over the short-context model's near-zero performance. For a developer navigating an unfamiliar codebase and asking questions like "where is the function that handles JWT token validation?" or "find the implementation of the rate limiter," a tool powered by this model could search across tens of thousands of tokens of code in a single forward pass, finding candidate functions without requiring pre-built indexing, embedding-based retrieval, or static analysis infrastructure. The 16K evaluation context (RepoQA's fixed size) is smaller than the model's 128K capacity; extending this use case to proportionally larger repository contexts would enable sub-file-granularity search across entire medium-sized projects in a single inference call, replacing multi-stage retrieval pipelines with end-to-end model-based search. The language-specific variance in the results (Java at 63.0% exact match vs. C++ at 3.0% for the 8B instruct model at threshold 0.8, per Table 4) means deployment should target the languages where retrieval is strongest, or include language-specific calibration to set appropriate confidence thresholds.

Synthetic training data generation for repository-scale code understanding tasks. The paper's bootstrapped instruction data generation pipeline (Section 2.2)—using a short-context model plus structured pretraining documents to produce multi-turn instruction data—is itself a practical tool that other teams can adopt immediately. A team building a code understanding dataset for fine-tuning can take their existing pretraining corpus, run the repository packing and dependency ordering algorithm, parse documents into functions/methods, and generate instruction-response pairs using any available instruct model (not necessarily long-context). The three task types (retrieval, explanation, implementation-from-context) provide a template for dataset construction that exercises different reasoning modes. Because the generation uses programmatic extraction for retrieval tasks (no model generation needed) and localized generation for explanation/implementation (short-context model suffices), the approach avoids the dependency on a long-context model that would otherwise make this data generation circular. The paper's evidence that this synthetic data improves instruct model performance on both long-context (RepoQA) and short-context (HumanEval synthesis) tasks suggests the data captures generally useful code understanding patterns. The open-source release of the Granite models under Apache 2.0 means teams can use the 128K instruct model itself as a generator for the next iteration, progressively improving data quality as described in the follow-up research direction above.

On-device long-context code models for air-gapped or privacy-sensitive development environments. The 3B model size is small enough for deployment on consumer hardware (laptops, edge servers) with quantization, and the 128K context window means the model can hold substantial repository context even in resource-constrained settings. For defense, finance, healthcare, or other sectors where source code cannot leave a secure environment, the Granite-128K-3B model provides repository-aware code completion and retrieval without network dependencies. The short-context preservation results (Table 5: 3B instruct model averages 30.9% on HumanEvalPack vs. 31.0% for the short-context version) mean the deployment does not sacrifice single-file code generation quality to gain long-context capability. The 4B-token continual pretraining budget (Section 2.1) is small enough that organizations with their own pretrained code models on proprietary codebases could replicate the recipe to extend their own models, using the paper's described methodology without needing to share sensitive training data. The specific hyperparameters (RoPE theta values per stage, batch size 32, 500 steps per stage, 10% short-document retention rate) provide a starting configuration that reduces the hyperparameter search burden for teams doing custom extensions.