ArXiv: 2510.00446

🎯 Pitch

LongCodeZip compresses code contexts by up to 5.6× with zero loss in task performance—and in some cases, even improves it—by using perplexity to preserve task-relevant structural dependencies that similarity-based retrieval misses. It achieves this without any training, making it a drop‑in efficiency boost for any code LLM facing long, real‑world inputs.

1. Executive Summary

This paper introduces LongCodeZip, a training-free, model-agnostic framework for compressing long code contexts before feeding them to code LLMs. Evaluated on Long Code Completion, Long Module Summarization, and RepoQA benchmarks using DeepSeek-Coder-6.7B, Qwen2.5-Coder-7B, Seed-Coder-8B, GPT-4o, and Claude-3.7-Sonnet, LongCodeZip employs a dual-stage strategy: coarse-grained compression—ranking function-level chunks by conditional perplexity with respect to the instruction and greedily selecting the most relevant—followed by fine-grained compression—segmenting retained functions into blocks via perplexity-based boundary detection, then solving a 0/1 knapsack optimization to select an optimal subset under an adaptive token budget. The framework achieves up to a 5.6× compression ratio without sacrificing task performance, matching or exceeding the no-compression baseline across tasks while reducing generation latency from 15.7s to 6.6s on code completion, and demonstrates strong cross-model generalization—a 0.5B compression model drives competitive downstream performance on larger generation models—establishing that semantic-level code compression can preserve task-critical dependencies that similarity-based retrieval misses, though only when the context contains information actually relevant to the instruction.

2. Context and Motivation

The Core Problem: Code LLMs Need Long Contexts, But Long Contexts Break Them

Modern software development increasingly demands that LLMs reason over entire codebases rather than isolated snippets. A developer asking an LLM to complete a function in a large project needs the model to understand class hierarchies defined elsewhere, configuration parameters set in other files, and utility functions imported across the repository. This means the model's input context—the concatenation of the instruction plus all relevant source files—can easily span tens of thousands of tokens, far beyond conventional LLM input lengths.

The paper identifies three distinct pain points that arise when these long code contexts meet current LLM architectures:

  1. Quadratic attention complexity produces latency cliffs. The standard transformer attention mechanism scales as O(n2)O(n^2) with sequence length nn (Section I, citing Vaswani et al., 2017). When a code context grows from 2,000 to 10,000 tokens, the attention computation doesn't just get 5× slower—it gets approximately 25× more expensive. For interactive developer tools where sub-second latency matters, this is unacceptable regardless of the underlying hardware.

  2. API costs explode with input length. Commercial LLM providers (the authors explicitly reference GPT-4 and Claude-3.7-Sonnet pricing models in Section I) charge primarily per input token. A single query that sends 10,000 tokens of repository context can cost dollars rather than cents, making repository-scale LLM usage economically infeasible for continuous integration pipelines or developer tools that process thousands of queries daily.

  3. The "lost in the middle" problem interacts destructively with code structure. Liu et al. (2023) demonstrated that LLMs struggle to utilize relevant information when it appears in the middle of long contexts. In code, this is particularly damaging because critical dependencies—a class definition used by a function, a type annotation referenced throughout a module—may appear anywhere in the input, not just at the beginning or end. When the model fails to "see" these dependencies, it generates code that looks syntactically plausible but is semantically broken: it hallucinates parameter types, invents missing methods, or silently ignores configuration values (Section II, Figure 1).

Even models that boast 128K token context windows (Section I cites Qwen2.5-Coder and CodeLlama as examples) don't escape this trap in practice. When a repository contains multiple large files and the conversation history accumulates, the 128K limit can still be exceeded, forcing truncation that clips out potentially critical information (Liu et al., 2023; Bogomolov et al., 2024).

Why Existing Solutions Fall Short for Code

The paper situates itself against three classes of prior approaches, each of which fails in code-specific ways:

General-Text Compression Methods Don't Understand Code Structure

Prompt compression techniques developed for natural language—the paper cites LLMLingua (Jiang et al., 2023), LongLLMLingua (Jiang et al., 2024), LLMLingua-2 (Pan et al., 2024), and Selective Context (Li et al., 2023)—operate at the token level. They prune individual tokens or sentences based on learned importance scores, perplexity heuristics, or data-distilled classifiers. This works adequately for prose, where removing a few adjectives or a parenthetical clause doesn't break anything fundamental.

In source code, however, token-level pruning is catastrophic (Section V.A reports LLMLingua achieves 1.5–8.7% accuracy on RepoQA, essentially random performance). Removing a single closing brace, a type annotation, or a semicolon can change program semantics entirely or make the remaining code unparseable by the downstream LLM. More subtly, code has non-local dependencies: the meaning of a variable reference at line 200 depends on an import statement at line 5, which might look "unimportant" to a token-level importance scorer but is actually load-bearing for the entire file.

The paper's experiments bear this out dramatically. On the Long Code Completion task (Table II) with Qwen2.5-Coder-7B, LLMLingua achieves only 21.56 ES and 5.40 EM at a 3.4× compression ratio—far below RAG-based methods (52.79 ES) and representing a catastrophic collapse relative to the no-compression baseline (56.36 ES). LLMLingua-2 does somewhat better (41.29 ES) but still trails substantially. The damage is structural: these methods treat code as unstructured text and destroy the syntactic coherence the downstream model depends on.

Retrieval-Augmented Generation Misses Implicit Dependencies

RAG-based approaches (Section II, citing RepoCoder, UniXCoder, and CodeBERT) reduce context by retrieving only the code snippets that appear "similar" to the task instruction, typically using embedding models like UniXCoder (Guo et al., 2022) with cosine similarity between the instruction embedding and candidate snippet embeddings.

The paper's key motivating insight—illustrated in Figure 1 with two concrete examples—is that lexical similarity is a poor proxy for code relevance. In the first example, completing a get_email_by_id function, RAG works well because the candidate context (Account class, get_account_by_id function) shares obvious surface-level similarity: overlapping function names, similar parameter names (user_id), and adjacent structural context. The embedding similarity correctly ranks these as highly relevant.

In the second example, completing a train_model function that needs to use configuration values from a Config class, RAG fails systematically. The Config class has minimal lexical overlap with train_model—different identifiers, different structural patterns, no shared substrings beyond generic programming keywords. Yet Config is the most critical piece of context because train_model needs config.lr, config.epochs, and other configuration attributes to set up the optimizer correctly. Without it, the model generates code that references undefined attributes or hardcodes incorrect values.

The paper quantifies this failure: on RepoQA (Table IV), RAG with function-level chunking achieves only 42.3–54.3% accuracy, while LongCodeZip exceeds 75–87%. The gap is not a small efficiency difference—it's the difference between unusable and production-ready. RAG misses the Config-class types of dependencies because embedding similarity cannot capture functional coupling (two components need each other to work correctly even though they look different) or abstract dependencies (type annotations, inheritance hierarchies, configuration contracts).

Code-Specific Compressors Are Either Too Coarse or Too Narrow

Prior code compression work—the paper specifically evaluates DietCode (Zhang et al., 2022) and SlimCode (Wang et al., 2024)—addresses code structure, but with different limitations:

DietCode uses static frequency-based filtering combined with CodeBERT attention heuristics to identify low-impact tokens for pruning. The problem is that its reliance on model-specific attention patterns makes it architecture-dependent: attention patterns learned by CodeBERT don't necessarily transfer to DeepSeek-Coder or Qwen2.5-Coder. The paper's results show DietCode achieving only 43.91 ES on Qwen2.5-Coder-7B for code completion (Table II) and 26.0% on RepoQA (Table IV), substantially below LongCodeZip. Additionally, DietCode was designed for compressing single functions, not repository-scale contexts with cross-file dependencies.

SlimCode applies rule-based token pruning using token types and program dependency graphs. While more principled than DietCode in its structural awareness, its rules are language-specific (the original implementation only supports Java; the authors reproduced a Python version using tree-sitter for fair comparison) and may not handle the diverse set of dependency patterns that span multiple functions and files. On RepoQA, SlimCode achieves only 34.0% with Qwen2.5-Coder-7B (Table IV), barely above random line removal (11.8%). The paper notes that both DietCode and SlimCode were "generally limited to function-level pruning or short code examples," leaving the long-context, multi-file scenario largely unaddressed.

The Missing Piece: Semantic Relevance Beyond Surface Similarity

The common thread in these failures is the inability to measure semantic relevance in a way that captures code-specific dependency structures. Similarity-based retrieval sees functions that share words. Token-level compression sees individual tokens with local importance. But code relevance often manifests as mutual information: given a task instruction (complete this function), which parts of the context actually help the model predict the correct code?

The paper introduces this perspective through the approximated mutual information (AMI) formulation (Equation 1, Section III-A), defined as:

AMI(c,q)=PPL(q)−PPL(q∣c)\text{AMI}(c, q) = \text{PPL}(q) - \text{PPL}(q \mid c)

where PPL(q∣c)\text{PPL}(q \mid c) is the conditional perplexity of the instruction qq given context cc, and PPL(q)\text{PPL}(q) is the unconditional perplexity. A high AMI score means that providing the context cc significantly reduces the model's uncertainty about what comes next in the instruction sequence—in other words, cc contains information the model needs to understand and execute the task. This formulation is the paper's core conceptual advance: it replaces the question "does this context look like the instruction?" (similarity) with "does this context help predict the instruction?" (relevance).

This framing directly addresses the Config-class scenario from Figure 1. The Config class has low lexical similarity to train_model, but it has high mutual information: knowing the configuration attributes reduces the model's perplexity when predicting the train_model completion because it clarifies what parameters are available. The AMI score captures this functional coupling even when surface features don't align.

How This Paper Positions Itself

The paper makes its positioning explicit in three ways:

First, it claims novelty as "the first framework specifically designed for long-context code compression" (Section VIII-B, final paragraph). Prior code compressors addressed single-function or short-example settings. Prior long-context compressors addressed natural language. LongCodeZip sits at the intersection: it handles long, multi-file code contexts using code-aware techniques (perplexity-based block boundary detection, function-level chunking) while remaining training-free and model-agnostic—no fine-tuning on the target LLM, no dependency on specific model architectures. This "plug-and-play" property (Section III introduction) is deliberately contrasted with DietCode's model-specific attention heuristics and soft-prompt methods that require training on the target model (Section VIII-B, citing Mu et al., 2023; Li et al., 2024).

Second, it frames the two-stage compression pipeline as addressing different failure modes of prior work. The coarse-grained stage (function-level selection by AMI ranking) solves RAG's lexical similarity limitation by using a signal that captures functional relevance. The fine-grained stage (perplexity-based block segmentation and knapsack selection) solves token-level compressors' structure-destroying behavior by operating on code-level semantic units while preserving syntactic coherence. The combination achieves a balance that neither approach alone can (Table VII ablation: removing fine-grained compression reduces ES by 1.45 points; replacing AMI ranking with similarity-based ranking reduces ES by 7.89 points—the coarse stage provides roughly 5× the impact of the fine stage, but both are necessary).

Third, it is explicitly situated as complementary to—not competing with—RAG and advanced retrieval methods. Section V.A notes that "RAG-based retrieval methods are complementary to our compression approach and could potentially be combined with our framework to further enhance performance." Section VIII.B (final paragraph) reiterates: "Unlike these approaches that are specifically designed for repository-level code completion, we propose a training-free code context compression technique that provides broader applicability across diverse long-context code tasks." LongCodeZip can, in principle, sit on top of a RAG pipeline: RAG retrieves candidate files, LongCodeZip compresses them before feeding to the LLM. The paper doesn't demonstrate this combination, but the architectural framing is clear. Table VI shows LongCodeZip outperforming advanced RAG methods (A3-CodGen, cAST, RepoGenix, RLCoder) on code completion, but this is presented as validating the compression approach, not as an argument against retrieval altogether.

The Broader Significance

The paper's motivation extends beyond the immediate problem of compressing long code contexts. There is an implicit argument about how code LLMs should be deployed in practice. The prevailing paradigm has been to extend context windows (train models to handle longer sequences) or to add retrieval systems on top (retrieve relevant snippets, concatenate, feed to LLM). LongCodeZip proposes a third path: compress what you already have, preserving semantic content while reducing token count.

This matters for three practical scenarios that the paper doesn't explicitly call out as motivations but which emerge from the experimental design:

  • Resource-asymmetric deployments: When the compression model is small (0.5B parameters, Table VIII) but the generation model is large (7B+ parameters or commercial API), the cost structure is highly asymmetric—expensive per-token generation, cheap per-token compression. Spending cheap compute on compression to save expensive compute on generation is economically rational, and the paper's cross-model transferability results (RQ3, Table VIII) validate that this works.

  • Latency-sensitive interactive tools: The efficiency analysis (Table IX) shows compression overhead of 2.58 seconds yielding a generation time reduction from 15.7s to 6.6s—a net savings of ~6.5 seconds per query. If this were an IDE autocomplete that fires hundreds of times per session, the cumulative savings are substantial.

  • The "long context is here, but not quite" regime: Even models with 128K token windows may still exceed their limits on real repositories. LongCodeZip provides a safety valve: if the concatenated context would overflow the window, compress to fit rather than truncating blindly.

The paper explicitly acknowledges that LongCodeZip has a fundamental limitation: it can only preserve what's in the context to begin with. If the context lacks information relevant to the instruction, or if the instruction is too ambiguous to align with any context segment, the method has nothing useful to select (Section VI.B). But for the common case where the repository does contain the needed information but it's buried in a sea of irrelevant code, LongCodeZip provides a principled way to find and preserve it.

3. Technical Approach

3.1 Reader orientation

LongCodeZip is a software system that takes a long chunk of source code (potentially tens of thousands of tokens spanning multiple files) plus a task instruction (e.g., "complete this function") and produces a shorter version of that source code that fits within a specified token budget while preserving the information most critical for the downstream LLM to perform the task correctly. It solves the problem that code LLMs struggle with long inputs—both in terms of computational cost and in terms of attention dilution—by identifying which parts of the context actually reduce the model's uncertainty about the instruction and keeping only those parts, using a two-stage coarse-to-fine pipeline that first filters out whole irrelevant functions and then prunes individual semantic blocks within the retained functions.

3.2 Big-picture architecture (diagram in words)

The system has five major components connected in a sequential pipeline:

  1. Function-Level Chunker — takes the raw long code context and splits it into individual function/class definitions, producing a list of self-contained code chunks. Its responsibility is ensuring that every piece of code the system considers is syntactically coherent rather than arbitrary slices.

  2. AMI Scorer and Ranker — for each function chunk, computes the approximated mutual information between that chunk and the task instruction using a small language model, then ranks chunks from most to least relevant. Its responsibility is the coarse-grained "what matters?" decision that similarity-based retrieval gets wrong.

  3. Budget-Constrained Function Selector — greedily takes the top-ranked functions until the accumulated token count reaches a coarse-grained budget threshold, replacing unselected functions with placeholder markers. Its responsibility is making the first major compression cut while preserving enough context for the fine stage to work with.

  4. Perplexity-Based Block Segmenter — for each retained function, scans line-by-line perplexity scores (computed by the same small LM) to identify semantic boundaries where perplexity spikes, splitting the function into a small number of coherent blocks. Its responsibility is finding code-level "paragraphs" that can be pruned individually without breaking syntax.

  5. Adaptive Budget Allocator and 0/1 Knapsack Block Selector — distributes the remaining token budget across retained functions proportionally to their importance, then within each function solves a dynamic programming optimization to pick the subset of blocks that maximizes total relevance within that function's allocation. Its responsibility is the fine-grained "what specifically to keep?" decision that maximizes information density in the final compressed output.

Information flows strictly forward: raw code → function chunks → ranked functions → top-N retained functions → segmented blocks → selected block subsets → compressed context. The compressed context is then concatenated with the task instruction and fed to the downstream generation LLM (which can be a different, larger model than the one used for compression).

3.3 Roadmap for the deep dive

  • First, the formal problem statement and the approximated mutual information (AMI) metric, since AMI is the mathematical foundation that every subsequent component depends on for relevance scoring.
  • Second, the coarse-grained compression stage (function chunking, AMI-based ranking, greedy selection), since this is the first major decision point and establishes which code enters the fine stage.
  • Third, the fine-grained compression stage in its three sub-parts: perplexity-based block boundary detection (how we split functions), adaptive budget allocation (how we distribute tokens across functions of varying importance), and 0/1 knapsack block selection (how we pick which blocks survive within each function).
  • Fourth, the compression model and its relationship to the generation model, since the cross-model transferability is a key design claim.
  • Fifth, the key hyperparameters and task-specific configurations, since these vary across code completion, summarization, and QA and encode important design tradeoffs.

This order follows the data pipeline from input to output and builds understanding cumulatively: each component's motivation and behavior depends on understanding what precedes it.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems and methods paper whose core idea is that context compression for code LLMs should be driven by semantic relevance measured through conditional perplexity rather than embedding similarity, and that a two-stage function-then-block selection process can preserve code structure while achieving aggressive compression ratios. The entire approach is training-free (no gradient updates to any model), model-agnostic (works with any code LLM that exposes token probabilities), and plug-and-play (the compressed output is standard text that any LLM can consume).

The Core Metric: Approximated Mutual Information (AMI)

The entire LongCodeZip framework is built on a single mathematical primitive: given a candidate piece of context $c$ (a function, a block of code) and a task instruction $q$, how much does $c$ reduce the model's uncertainty about $q$? This is formalized through approximated mutual information, defined in Equation 1 of the paper.

The starting point is the standard definition of perplexity. For any sequence of tokens $q = \{q_1, q_2, \ldots, q_N\}$, the conditional perplexity given context $c$ is:

PPL(q∣c)=exp⁡(−1N∑i=1Nlog⁡P(qi∣q<i,c))\text{PPL}(q \mid c) = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \log P(q_i \mid q_{<i}, c)\right)

where $N$ is the number of tokens in the instruction, $q_i$ is the $i$-th token, $q_{<i}$ is the sequence of tokens before position $i$ (the prefix), and $P(q_i \mid q_{<i}, c)$ is the language model's predicted probability for token $q_i$ given both the preceding instruction tokens and the context $c$.

What it computes: For each position in the instruction sequence, the model assigns a probability to the token that actually appears there, conditioned on all previous instruction tokens AND the context. These probabilities are converted to log-space (more numerically stable), averaged, negated, and exponentiated. The result is a single positive number measured in "effective vocabulary size" units—a perplexity of 50 means the model is as uncertain as if it were choosing uniformly among 50 equally likely options at each step. Lower perplexity means the model is more confident about what comes next, which implies it has more information about the task.

The unconditional perplexity $\text{PPL}(q)$ is computed identically but without the context $c$:

PPL(q)=exp⁡(−1N∑i=1Nlog⁡P(qi∣q<i))\text{PPL}(q) = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \log P(q_i \mid q_{<i})\right)

What it computes: The model's baseline uncertainty about the instruction when it sees only the instruction itself, with no additional code context. This is the "no-information" reference point.

The approximated mutual information between context and instruction is then simply the difference:

AMI(c,q)=PPL(q)−PPL(q∣c)\text{AMI}(c, q) = \text{PPL}(q) - \text{PPL}(q \mid c)

where $\text{PPL}(q)$ is the unconditional perplexity and $\text{PPL}(q \mid c)$ is the conditional perplexity given context $c$.

What it computes: A single scalar score measuring how much $c$ reduces the model's perplexity about $q$. If $c$ provides no useful information (conditional perplexity equals unconditional perplexity), AMI is zero—the context doesn't help predict the instruction at all. If $c$ provides a lot of information (conditional perplexity much lower than unconditional), AMI is large and positive—the context significantly clarifies what the instruction is about. The maximum possible AMI is $\text{PPL}(q) - 1$, though this is never achieved in practice since no context makes the model perfectly certain.

Why this form: The paper uses perplexity difference rather than alternatives like cosine similarity between embeddings for several reasons. First, perplexity captures functional relevance rather than surface similarity. A Config class that defines self.lr and self.epochs may have zero lexical overlap with a train_model function that needs those values, but the LM's perplexity when predicting tokens like config.lr or optimizer = torch.optim.AdamW(lr=config.lr) will drop substantially when the Config class is in context—because the model has learned during pretraining that training functions often reference configuration objects. The AMI score picks up on this learned co-occurrence structure that embedding similarity cannot.

Second, perplexity-based scoring is training-free and model-agnostic. Any language model that exposes token-level log-probabilities can be used as the "compression model" to compute AMI scores. The paper's cross-model experiments (Table VIII) deliberately use different models for compression (computing AMI) and generation (producing the final output) to demonstrate that the AMI signal transfers across architectures—the "understanding" of what context is relevant appears to be a fairly universal property of pretrained code LMs, not something specific to fine-tuning or architecture choice.

Third, the subtraction form $\text{PPL}(q) - \text{PPL}(q \mid c)$ has an information-theoretic interpretation as an approximation to pointwise mutual information. True mutual information would require marginalizing over all possible instructions $q$, which is computationally infeasible. The per-instruction approximation used here is cheap (requires only two forward passes of the compression model per candidate $c$: one with context, one without) and provides a ranking signal that correlates well with actual downstream usefulness.

A subtle operational detail: the paper computes AMI by conditioning the LM on $c$ and measuring perplexity on $q$—not the other way around. This is because the task instruction is typically much shorter than the context (tens of tokens vs. thousands), so computing $\text{PPL}(c \mid q)$ would be far more expensive. The chosen direction is sufficient because information-theoretically, if $c$ helps predict $q$, then $c$ and $q$ share information content regardless of which direction the probability is measured in. This is an approximation—$\text{PPL}(q) - \text{PPL}(q \mid c)$ is not strictly symmetric—but it works well in practice as a relevance ranking signal.

Stage 1: Coarse-Grained Compression via Function Selection

The coarse-grained stage is responsible for the largest compression gains by eliminating entire functions that are irrelevant to the task. It operates in three sequential steps, each of which is conceptually simple but whose combination is critical to the overall performance. The ablation in Table VII shows that this stage accounts for roughly 80% of LongCodeZip's advantage over similarity-based methods (ES drops by 7.89 points when AMI ranking is replaced with similarity ranking, and by 17.79 points when replaced with random ranking).

Step 1: Function-Level Chunking

The input long code context is split along function and class boundaries. The paper uses tree-sitter (a parser generator and incremental parsing library) to identify these boundaries across multiple programming languages—the RepoQA benchmark includes Python, Java, JavaScript, TypeScript, Rust, Go, and C++, all of which tree-sitter supports with language-specific grammars. Each extracted chunk is a complete function definition, class definition, or top-level code block that is syntactically self-contained.

Why function boundaries: Functions are the natural unit of code modularity. A function encapsulates a coherent piece of logic: it has well-defined inputs (parameters), well-defined outputs (return values), and internal state that doesn't leak to other functions except through explicit interfaces. When a downstream LLM is asked to complete a function, understanding another function typically requires understanding its entire definition—its signature (to know what arguments it takes), its body (to know what it does), and its docstring (to know its contract). Splitting functions mid-body would produce fragments that are ambiguous or misleading. The paper also notes in Section III-C that function-level chunking "ensures that retained code segments are both syntactically valid and semantically self-contained, which is essential for preserving program integrity."

Classes are treated similarly to functions: a class definition with all its methods forms a single chunk. This is important because methods within a class typically share instance variables (self.x in Python, this.x in Java/JavaScript) defined in the constructor or other methods, and splitting them apart would break these internal dependencies. The paper doesn't explicitly discuss inheritance hierarchies or cross-file class relationships in the chunking step—these are handled implicitly by the AMI scoring, which captures functional dependencies regardless of whether they cross chunk boundaries.

Step 2: Instruction-Aware Relevance Ranking

Each function chunk is scored against the task instruction using the AMI metric defined above. The compression LM computes $\text{PPL}(q)$ once (the unconditional perplexity of the instruction alone) and $\text{PPL}(q \mid c_i)$ for each candidate function $c_i$ (the conditional perplexity of the instruction given that function as context). The AMI for function $i$ is:

AMIi=PPL(q)−PPL(q∣ci)\text{AMI}_i = \text{PPL}(q) - \text{PPL}(q \mid c_i)

Functions are then sorted in descending order of AMI score. The highest-scoring function is the one whose presence most dramatically reduces the model's uncertainty about what the instruction is asking.

What "high AMI" means operationally: If the instruction is "Complete the following function: def train_model(model, dataloader, config: Config): ...", the LM's unconditional perplexity on this instruction might be high because the model has many equally plausible continuations (it doesn't know what Config contains, what optimizer to use, etc.). When the Config class definition is provided as context, the LM "sees" that Config has attributes lr, epochs, beta1, beta2, weight_decay, and its perplexity on the instruction drops—because now it knows what the completion likely needs to reference. The AMI score for Config would be large. In contrast, a utility function like cosine_similarity might have near-zero AMI if the instruction gives no hint that similarity computation is needed—its presence doesn't help predict the instruction.

Why not similarity: The paper explicitly contrasts this with embedding-based similarity. If the compression were based on cosine similarity between instruction embeddings and function embeddings (as RAG does), cosine_similarity might score high if the function names or docstrings share words with the instruction, while Config might score low because "configuration" vocabulary doesn't overlap with "training" vocabulary. The AMI approach inverts this: it asks the model "does this help?" rather than "does this look similar?", which captures the functional coupling that similarity misses.

Computational note: Computing AMI for each function requires one forward pass of the compression LM per function (to get the log-probabilities of the instruction tokens given that function as context). For a context with 100 functions, this means roughly 100 forward passes, each processing the function tokens plus the instruction tokens. The paper's efficiency analysis (Table IX) reports a total compression time of 2.58 seconds for the entire pipeline—function scoring is the dominant cost. However, this cost is incurred once per query and can be amortized if the same context is reused across multiple instructions. The paper also shows (Table VIII) that a 0.5B compression model achieves nearly the same ranking quality as a 7B model, so using a smaller model significantly reduces this cost (the 2.58s figure is for Qwen2.5-Coder-7B as the compression model; a 0.5B model would be substantially faster).

Step 3: Budget-Constrained Function Selection

Given the ranked list of functions and a coarse-grained token budget $B_{\text{coarse}}$, the system greedily selects functions from highest AMI to lowest until adding the next function would exceed the budget. Formally:

Bcoarse=BRfineB_{\text{coarse}} = \frac{B}{R_{\text{fine}}}

where $B$ is the final target token budget (e.g., 2,000 tokens for code completion, 5,000 for summarization) and $R_{\text{fine}}$ is a configurable "fine-grained compression ratio" that controls how much of the final budget is reserved for the fine stage. Since the fine stage will further prune within selected functions, the coarse stage should select more tokens than the final budget allows—$R_{\text{fine}}$ determines how much more.

For code completion, $R_{\text{fine}} = 0.8$, meaning the coarse stage selects up to $2{,}000 / 0.8 = 2{,}500$ tokens worth of functions. The fine stage then compresses these 2,500 tokens down to the final 2,000 token budget. For code summarization, $R_{\text{fine}} = 0.3$, so the coarse stage selects up to $5{,}000 / 0.3 \approx 16{,}667$ tokens—a much larger coarse budget because summarization needs to understand broader context across modules before the fine stage can aggressively prune. For RepoQA, $R_{\text{fine}} = 1.0$, meaning no fine-grained compression and the coarse budget equals the final budget—this is because RepoQA requires preserving entire functions (the task is to retrieve and reproduce exact function code), so intra-function pruning would be counterproductive.

Greedy selection semantics: The paper uses greedy selection (take the highest-AMI functions in order until budget is exhausted) rather than solving a knapsack optimization at the function level. This is a pragmatic choice: function token counts vary widely (a one-line getter might be 20 tokens; a complex class with 10 methods might be 5,000 tokens), and an exact knapsack solution would require evaluating $2^N$ subsets. Greedy selection is optimal for the knapsack problem when items are sorted by value-to-weight ratio and the budget is large relative to individual item sizes—both conditions hold here since AMI scores are the "value" and the ratio AMI/token_count is what greedy selection effectively optimizes.

Placeholder strategy: Functions not selected are replaced with placeholder markers (e.g., comment markers or ellipses) in the compressed context. This is a deliberate design choice: the compressed context preserves the global structure of the original codebase (function A appears before function B, class C appears between them) even though the bodies of unselected functions are elided. The paper argues (Section III-C) that this "preserves the global structure while reducing overall context length." This matters because the downstream LLM may rely on relative positioning—if a selected function train_model typically appears after Config in a file, preserving that ordering helps the model understand the logical flow. The placeholders are essentially zero-information markers that signal "there was something here that we removed."

Stage 2: Fine-Grained Compression via Block Selection

After the coarse stage selects a set of relevant functions, the fine stage operates within each selected function to further prune content that is less critical while preserving content that carries the most information for the task. This stage is itself composed of three sub-steps: block segmentation, adaptive budget allocation, and knapsack-based block selection.

Step 1: Perplexity-Based Block Segmentation

The challenge in intra-function compression is identifying prunable units that are smaller than whole functions but larger than individual tokens or lines. Pruning individual tokens breaks syntax (as LLMLingua catastrophically demonstrates). Pruning individual lines is better but still risks splitting tightly coupled consecutive lines that form a logical unit—for example, a multi-line condition or a loop body. The paper's solution is to segment each function into semantic blocks using a perplexity-based boundary detection algorithm.

The algorithm treats each line of code as the smallest atomic unit and computes the unconditional perplexity of each line using the compression LM:

PPL(linej)=exp⁡(−1Lj∑k=1Ljlog⁡P(tk∣t<k))\text{PPL}(\text{line}_j) = \exp\left(-\frac{1}{L_j} \sum_{k=1}^{L_j} \log P(t_k \mid t_{<k})\right)

where $L_j$ is the number of tokens in line $j$, $t_k$ is the $k$-th token of the line, and the probabilities are computed with no additional context beyond the line itself (or equivalently, with only the preceding tokens within the line).

What this captures: A line that is "unsurprising" given its own prefix—for instance, a straightforward assignment like self.lr = lr—will have low perplexity because the model can easily predict the tokens. A line that begins a new logical concept—for instance, the first line of a new method or a try: block—will have high perplexity because the model, seeing only preceding tokens within that line, doesn't expect this shift in semantics. The key insight, adapted from the Meta-Chunking technique for natural language (Zhao et al., 2024, cited in Section III-D), is that within a semantically coherent region, perplexity tends to decrease as context accumulates, while a sharp local increase signals the start of a new semantic unit.

The boundary detection rule: a line $j$ is marked as a block boundary if:

PPL(linej)−PPL(linej−1)>α⋅σ\text{PPL}(\text{line}_j) - \text{PPL}(\text{line}_{j-1}) > \alpha \cdot \sigma

where $\sigma$ is the standard deviation of perplexity scores across all lines in the function, and $\alpha$ is a sensitivity parameter. The paper doesn't specify the exact value of $\alpha$ in the main text—it's described qualitatively as "a sharp local increase" that "exceeds that of its neighbors by at least $\alpha$ times of the standard deviation over all lines." In practice, $\alpha$ would be tuned on the small held-out set mentioned in Section IV-E.

What happens to the blocks: Consecutive lines between two detected boundaries form a single block. Each block is a contiguous span of code that—according to the LM's perplexity signal—forms a coherent semantic unit. Figure 4 provides a concrete illustration: within the evaluate_blind function, the perplexity distribution shows spikes at specific lines, and the resulting blocks separate major structural elements (the prefix/suffix extraction, the action lookup, the code execution branch) into distinct groups.

Why perplexity-based and not syntax-based: The paper could have used AST-based segmentation (split at statement boundaries, method boundaries, etc.), but this over-segments code into units that are individually too small for meaningful AMI scoring. A single if statement might span 3 lines and form a logical whole; splitting it into condition, body opening, and body closing would destroy the coherence. Perplexity-based segmentation adaptively groups tightly coupled lines based on the LM's learned understanding of code semantics—the same model that will later be used for generation. This is more aligned with how the downstream LLM "reads" code than a purely syntactic parse.

A subtlety about "unconditional" perplexity of a line: The paper's description says the perplexity is computed "as in (3)," which is the unconditional perplexity formula with no context beyond the line's own prefix tokens. This means the LM doesn't see surrounding lines when computing line-level perplexity. This is intentional: if the LM could see the previous line, it would be less surprised by a line that naturally follows—but the goal is precisely to detect when a line is "surprising given its local context," and that local context is the line itself. The standard deviation $\sigma$ is computed across all lines within a single function, so the threshold is function-relative—what counts as a "spike" in a function with highly variable perplexity is different from what counts in a function with consistently low perplexity.

Step 2: Adaptive Budget Allocation Across Functions

The coarse stage selected functions based on their AMI scores, but those scores also encode relative importance that should influence how much of the fine-grained budget each function receives. A function with very high AMI (e.g., the Config class for the train_model example) should retain more of its internal detail than a function with moderate AMI that was selected because the coarse budget was generous. The paper introduces an adaptive allocation mechanism summarized in Algorithm 1.

First, the retained functions are partitioned into two sets:

  • $F_{\text{small}}$: functions shorter than five lines. These are kept in their entirety because compressing them further offers negligible savings and risks losing critical short definitions (e.g., one-line accessor methods, decorator applications).
  • $F_{\text{large}}$: all other functions. These are subject to fine-grained compression.

The baseline retention ratio for large functions—what fraction of their tokens would be kept if importance were uniform—is computed as:

Rbase=B−∑j∈FsmallTj∑k∈FlargeTkR_{\text{base}} = \frac{B - \sum_{j \in F_{\text{small}}} T_j}{\sum_{k \in F_{\text{large}}} T_k}

where $B$ is the final token budget, $T_j$ is the token count of small function $j$, and $T_k$ is the token count of large function $k$.

What it computes: The total budget remaining after keeping all small functions intact, divided by the total number of tokens across all large functions. If the large functions collectively contain 10,000 tokens and the remaining budget is 4,000 tokens, $R_{\text{base}} = 0.4$—each large function would, under uniform allocation, retain 40% of its tokens. This is the "fair share" before importance adjustment.

For each large function $f_i$, its AMI score from the coarse stage is first min-max normalized across all retained functions:

AMInorm,i=AMIi−min⁡j(AMIj)max⁡j(AMIj)−min⁡j(AMIj)\text{AMI}_{\text{norm}, i} = \frac{\text{AMI}_i - \min_j(\text{AMI}_j)}{\max_j(\text{AMI}_j) - \min_j(\text{AMI}_j)}

This maps AMI scores to $[0, 1]$, where the most important retained function has normalized AMI = 1 and the least important retained function has normalized AMI = 0.

A biased retention ratio is then computed as:

Rbiased,i=Rbase⋅(1+β⋅(2×AMInorm,i−1))R_{\text{biased}, i} = R_{\text{base}} \cdot \left(1 + \beta \cdot (2 \times \text{AMI}_{\text{norm}, i} - 1)\right)

where $\beta \geq 0$ is the importance sensitivity parameter (set to 0.5 across all tasks in the paper).

What it computes: The term $(2 \times \text{AMI}_{\text{norm}, i} - 1)$ maps the normalized AMI to $[-1, 1]$: the most important function gets +1, the least important gets -1, and a function at the median gets 0. Multiplying by $\beta = 0.5$ and adding 1 produces a multiplier in $[0.5, 1.5]$ applied to the baseline rate. So the most important large function gets up to $1.5 \times R_{\text{base}}$ retention (50% more than uniform), and the least important gets as low as $0.5 \times R_{\text{base}}$ retention (50% less than uniform). When $\beta = 0$, all functions get exactly $R_{\text{base}}$—the adaptive allocation is effectively disabled, which the ablation (Table VII, "w/o Adaptive Budget Allocation") confirms hurts performance by 2.34 ES points.

Clamping and rescaling: The biased rates are clamped to $[0, 1]$ (you can't retain negative tokens or more than 100% of a function's tokens) and then globally rescaled to exactly satisfy the total budget constraint:

Ri=Rbiased,i⋅Blarge∑jRbiased,j⋅TjR_i = R_{\text{biased}, i} \cdot \frac{B_{\text{large}}}{\sum_j R_{\text{biased}, j} \cdot T_j}

where $B_{\text{large}}$ is the portion of the total budget $B$ allocated to large functions (total budget minus tokens for small functions). This rescaling ensures that the sum of $R_i \times T_i$ across all large functions exactly equals $B_{\text{large}}$. The relative proportions between functions are preserved from the biased step; the rescaling only adjusts the absolute level so the constraint is met.

Why adaptive allocation matters: Without this mechanism, a 5,000-token function and a 200-token function would both receive the same retention ratio (say, 40%), even if the 200-token function is much more important. The 5,000-token function would consume $0.4 \times 5000 = 2000$ tokens of the budget while the 200-token function gets only $0.4 \times 200 = 80$ tokens. With adaptive allocation biased toward importance, the more important function (whether large or small) gets a higher retention ratio, directing the limited budget toward the content that the AMI scores indicate is most critical.

The paper sets $\beta = 0.5$ for all tasks based on the held-out tuning set (Section IV-E). This is a moderate bias—strong enough to measurably improve performance (2.34 ES points in ablation) but not so aggressive that unimportant retained functions are completely starved of tokens (they still retain some content, which provides context continuity). A larger $\beta$ would allocate nearly all tokens to the top-ranked function, starving others; a smaller $\beta$ would approach uniform allocation and lose the benefit of importance-aware distribution.

Step 3: 0/1 Knapsack Block Selection

Within each large function $f_i$, the system now has a token budget $B_i = R_i \times T_i$ and a set of blocks $\{b_1, b_2, \ldots, b_M\}$ produced by the perplexity-based segmentation. Each block $b_j$ has a "value" (its normalized AMI score relative to the instruction) and a "weight" (its token count). The problem is to select a subset of blocks whose total weight does not exceed $B_i$ while maximizing the sum of values.

Why this is a knapsack problem: Some blocks contain essential information (high AMI, moderate token count)—these are high-value items. Some blocks are verbose but low-information (low AMI, high token count)—these are low-value items. The knapsack formulation naturally handles the tradeoff: pick the blocks with the highest value-per-token ratio first, then fill remaining budget with the next-best blocks. This is fundamentally different from line-level selection (which would treat every line independently and could produce syntactically broken output) because blocks are coherent units that can be removed or kept as wholes.

The paper also allows for a user-defined preserved set $P$—blocks that are always kept regardless of their AMI score. Algorithm 2 shows that the budget is first reduced by the token count of all preserved blocks: $B_{\text{remain}} = \max(0, B_i - \sum_{j \in P} T_j)$. The remaining budget is then allocated via dynamic programming over the non-preserved blocks. The paper doesn't specify what goes into $P$ in the main experiments—it could include function signatures, class headers, or other structural elements that must be present for the code to parse correctly. The architecture allows for this flexibility without requiring it.

The dynamic programming solution to the 0/1 knapsack problem is standard: for budget $W = 0, 1, \ldots, B_{\text{remain}}$ and items $j = 1, \ldots, |K|$ (where $K$ is the set of non-preserved blocks), compute:

dp[j][W]=max⁡(dp[j−1][W],dp[j−1][W−Tj]+AMIj if W≥Tj)dp[j][W] = \max(dp[j-1][W], dp[j-1][W - T_j] + \text{AMI}_j \text{ if } W \geq T_j)

The optimal subset is recovered by backtracking through the DP table. The time complexity is $O(|K| \times B_{\text{remain}})$, which is manageable because both the number of blocks per function and the per-function budget are modest (a typical function might have 5–15 blocks and a budget of 100–500 tokens, measured in block token counts rather than individual tokens). The DP operates on block-level granularity—the "weight" of a block is its token count as an integer, and the budget is in tokens—so the DP table size is $|K| \times B_i$, which for a 200-token function budget and 10 blocks gives a 10 × 200 = 2,000 cell DP table, trivially solved.

Why DP and not greedy: Unlike the function-level selection (which uses greedy), the block-level selection uses exact dynamic programming. This is feasible because the per-function problem is small, and exact optimization matters more at this stage: the fine-grained stage is operating near the final budget constraint, so suboptimal block choices directly waste tokens that could have been used for higher-value blocks. At the function level, greedy works well because the budget is larger relative to individual function sizes, making the greedy approximation close to optimal. At the block level, the budget is tight relative to block sizes, making exact optimization more valuable.

AMI scoring for blocks: The paper specifies that blocks inherit their AMI scores from the function they belong to (the function-level AMI is used as the block's value). This is a simplification: computing per-block AMI would require separate forward passes of the compression LM for each block, which would multiply the compression cost by the number of blocks (potentially 5–15× more expensive). Using function-level AMI for all blocks within a function means the knapsack selection within a function is effectively choosing blocks to maximize the token count of represented functions, weighted by function importance—but since all blocks from the same function have the same AMI value, the knapsack within a single function actually selects blocks to maximize total tokens kept subject to budget, which reduces to keeping blocks in arbitrary order until the budget is exhausted. This appears to be an oversight in the paper's description—if all blocks in a function have identical AMI values, the knapsack doesn't discriminate between them based on relevance.

A more nuanced reading: the paper may be computing block-level AMI individually (the AMI of a block $b_j$ being $\text{PPL}(q) - \text{PPL}(q \mid b_j)$), in which case the knapsack within a function genuinely optimizes relevance-weighted selection. The ablation showing that knapsack selection outperforms random line selection by 2.48 ES (Table VII) supports this interpretation—if all blocks had equal AMI, random selection should perform identically to knapsack selection. The most likely implementation is that the AMI for each block is computed by conditioning the compression LM on that specific block (not the whole function) and measuring the reduction in instruction perplexity. The DP then optimizes over blocks with potentially different AMI values, capturing the intuition that some parts of a function (the signature and key logic) are more informative than others (logging statements, boilerplate error handling).

The Compression Model and Cross-Model Transferability

A critical design decision is which model computes the AMI scores. The paper uses the same model architecture for compression as for downstream generation in the main experiments (e.g., Qwen2.5-Coder-7B for both), but explicitly designs the framework to be model-agnostic—any code LM with accessible token log-probabilities can serve as the compression model, and it need not match the generation model.

The cross-model experiments (Table VIII) validate this claim systematically:

  • Using DeepSeek-Coder-6.7B for compression and Qwen2.5-Coder-7B for generation achieves 56.55 ES (vs. 57.55 when using Qwen2.5-Coder-7B for both).
  • Using Qwen2.5-Coder-0.5B for compression and DeepSeek-Coder-6.7B for generation achieves 61.12 ES (vs. 60.58 when using DeepSeek-Coder-6.7B for both).
  • The average ES across all cross-model pairs ranges from 59.54 to 60.58—a spread of only ~1 point, despite model sizes ranging from 0.5B to 8B parameters and release dates spanning 2023–2025.

Why this works: The AMI ranking signal appears to capture a fairly universal property of code—what information is relevant for understanding a programming task—that transfers across model architectures and training procedures. A 0.5B model trained on similar code data learns roughly the same co-occurrence patterns as a 7B model, just with higher baseline perplexity. Since AMI is a difference in perplexity rather than an absolute level, the baseline shift largely cancels out, leaving a relevance ordering that is consistent across scales.

This cross-model transferability has major practical implications. The compression model can be a small, fast model (0.5B parameters, negligible GPU memory, fast inference) while the generation model can be a large, expensive model (7B+ open-source or commercial API). The paper's efficiency analysis (Table IX) shows compression overhead of 2.58 seconds for Qwen2.5-Coder-7B, but this would be substantially lower for a 0.5B model—the paper explicitly suggests that using a lightweight model "will significantly reduce compression time and memory overhead, making our approach particularly suitable for resource-constrained scenarios." The compression memory overhead for the 7B model is reported as "Base + 0.69 GB," meaning 0.69 GB beyond the 28.37 GB base model parameters—this too would shrink for smaller compression models.

Key Hyperparameters and Task-Specific Configurations

The paper's experiments use task-specific hyperparameter configurations, tuned on a small held-out set that does not overlap with the test data (Section IV-E). The three configurations encode different tradeoffs between compression aggressiveness and information preservation:

Code Completion (B = 2K, R_fine = 0.8, β = 0.5):

  • Token budget $B$ = 2,000 — code completion tasks typically need focused context (the function being completed plus its direct dependencies), not the entire file. The average uncompressed context is 9,328 tokens (Table I), so a 2K budget represents roughly a 4.7× compression target.
  • Fine-grained ratio $R_{\text{fine}}$ = 0.8 — the coarse stage selects up to $2000 / 0.8 = 2500$ tokens, giving the fine stage a modest 20% pruning headroom. This is appropriate because completion tasks are sensitive to precise details within retained functions (parameter types, attribute names), so aggressive intra-function pruning risks removing critical tokens.
  • Importance parameter $\beta$ = 0.5 — moderate bias toward important functions, consistent across all tasks.

Code Summarization (B = 5K, R_fine = 0.3, β = 0.5):

  • Token budget $B$ = 5,000 — summarization requires understanding broader module-level context to produce accurate descriptions. The average uncompressed context is 10,810 tokens (Table I), so a 5K budget represents a roughly 2.2× compression target—which matches the achieved ratios in Table III (1.7–3.5× depending on model).
  • Fine-grained ratio $R_{\text{fine}}$ = 0.3 — the coarse stage selects up to $5000 / 0.3 \approx 16667$ tokens, a very large coarse budget that keeps most functions before fine-grained pruning. This is because summarization benefits from seeing the overall structure of the module (all functions, their relationships) before the fine stage aggressively prunes within each function. The low $R_{\text{fine}}$ means coarse selection is permissive while fine selection does the heavy lifting.
  • This configuration produces lower compression ratios (1.7–3.5×) than code completion because summarization genuinely needs more context to be effective.

RepoQA (B = 2K, R_fine = 1.0, β = 0.5):

  • Token budget $B$ = 2,000 — RepoQA is a retrieval-like task where the model must locate and reproduce an entire function. The average uncompressed context is 11,525 tokens (Table I), so a 2K budget is aggressive (~5.8×).
  • Fine-grained ratio $R_{\text{fine}}$ = 1.0 — the coarse budget equals the final budget exactly, meaning no fine-grained compression is applied. This is because RepoQA requires returning complete function bodies (the evaluation metric is BLEU between generated and target functions), so intra-function pruning would destroy the very information the task needs. The compression relies entirely on the coarse stage's function selection.
  • The achieved compression ratios in Table IV (4.5–5.3×) confirm that function-level selection alone can aggressively compress while preserving task performance—LongCodeZip achieves 75.3–87.2% accuracy vs. 38.3–86.0% no-compression baselines, and for GPT-4o and Claude-3.7-Sonnet, it actually exceeds the no-compression baseline (88.9% vs. 87.8% for GPT-4o, 88.9% vs. 89.7% for Claude-3.7-Sonnet in Table V—the compression paradoxically improves performance on Claude-3.7-Sonnet for RepoQA).

Why task-specific tuning matters: The three tasks represent different points on a spectrum from "precise local context is critical" (completion) to "broad structural understanding is critical" (summarization) to "one complete function is the answer" (RepoQA). The budget $B$ and fine-grained ratio $R_{\text{fine}}$ encode these differences: completion uses a tight budget with modest fine pruning; summarization uses a generous budget with aggressive fine pruning; RepoQA uses budget with zero fine pruning. The importance parameter $\beta$ is consistently 0.5, suggesting this value is relatively task-insensitive and primarily depends on the AMI score distribution across functions rather than the downstream task type.

The paper doesn't provide a systematic sweep of these hyperparameters or analyze sensitivity. The values were determined on a held-out set, but the ranges explored and the sensitivity of performance to each parameter aren't reported. This is a limitation—a practitioner wanting to apply LongCodeZip to a new task would need to perform their own tuning, and the paper doesn't provide guidance on what ranges are reasonable or which parameters interact strongly. The ablation study (Table VII) tests removing components entirely but doesn't vary $\beta$, $R_{\text{fine}}$, or $B$ within a task to show sensitivity curves.

4. Key Insights and Innovations

Innovation 1: Replacing "Is This Similar?" with "Does This Help?" as the Foundational Criterion for Code Context Selection

The paper's single most important conceptual move is abandoning lexical similarity as the relevance metric for code context and replacing it with a causal, information-theoretic criterion: does this context reduce the model's uncertainty about what the instruction is asking? This is measured through approximated mutual information (AMI), defined as the drop in instruction perplexity when the candidate context is provided.

This is not merely a new scoring function—it is a fundamentally different question. Similarity-based retrieval (RAG with UniXCoder or CodeBERT embeddings, cited in Section II) asks: "Does this code snippet contain words or patterns that look like words or patterns in the instruction?" This works when overlap is high—the get_email_by_id/get_account_by_id example in Figure 1—but fails systematically when the relationship is functional rather than lexical. A configuration class and a training function that uses it share no vocabulary, yet one cannot be understood without the other. Prior work had no mechanism for detecting this dependency, because embedding spaces encode what text looks like, not what it enables.

The AMI formulation answers a different question: "Given what this language model learned during pretraining about how code components relate, does seeing this context change the model's predictions about the instruction?" A code LM trained on millions of repositories has internalized that Config classes supply parameters to train_model functions—not through explicit annotation, but through the statistical co-occurrence patterns absorbed during next-token prediction. When the Config class is in context, the model's perplexity on the instruction drops because it anticipates tokens like config.lr or optimizer = AdamW(lr=config.lr) that would otherwise be surprising. The AMI score surfaces this latent knowledge.

The ablation study (Table VII) quantifies the magnitude of this conceptual shift: replacing AMI-based function ranking with similarity-based ranking drops ES by 7.89 points (from 57.55 to 49.66), and replacing it with random ranking drops ES by 17.79 points (to 39.76). These are not marginal improvements—the ranking criterion alone accounts for the majority of LongCodeZip's advantage over baselines. The coarse-grained stage provides roughly 5× the impact of the fine-grained stage in the ablation, and within the coarse stage, the scoring function (AMI vs. similarity vs. random) is the dominant factor.

This insight has significance beyond the specific mechanism. It establishes a principle that should generalize: for structured artifacts with latent functional dependencies (code, but also legal documents, mathematical proofs, circuit designs), surface-level similarity is a weak signal, and model-internal measures of predictive utility are stronger. The approach is training-free and model-agnostic—any LM with accessible log-probabilities can serve as the compressor—meaning it can be adopted immediately for any domain where a pretrained model captures the relevant dependency structure. The paper demonstrates this transferability explicitly (Table VIII): a 0.5B compression model produces AMI rankings that are nearly as effective for a 7B generation model as the 7B model's own rankings, confirming that the dependency knowledge reflected in AMI scores is a fairly universal property of code pretraining rather than something architecture-specific.

The key diagnostic move is subtle but powerful: the paper doesn't try to engineer a better similarity function for code (the path prior work took). Instead, it offloads relevance judgment to the pretrained model itself, treating the LM not as the object being optimized but as an oracle whose internal expectations encode all the dependency knowledge needed. This is a conceptual pivot from "build a better retriever" to "ask the model what it already knows about what belongs together." The fact that it works—and works dramatically better than engineered similarity functions—suggests that the dominant paradigm of retrieval-augmented generation (embed, score by cosine similarity, concatenate) is fundamentally limited for code and that model-introspective signals deserve a central role in context selection systems going forward.

Innovation 2: The Two-Stage Coarse-to-Fine Architecture as a Deliberate Solution to Code's Dual Compression Challenge

Code presents a compression challenge that natural language does not: you need to remove both entire irrelevant components and internal redundancy within relevant components, and these two operations require fundamentally different granularities and decision procedures. Prior code compressors (DietCode, SlimCode) either operated at a single granularity (token-level or rule-based pruning within individual functions) or were designed for short single-function contexts. General text compressors (LLMLingua) operated at token granularity and destroyed code structure. RAG operated at snippet granularity but missed functional dependencies. No prior system attempted the hierarchical combination that LongCodeZip introduces: function-level selection for the big structural cuts, followed by block-level selection for the fine detail preservation.

What makes this architecture intellectually distinctive is not the individual components—function chunking, AMI scoring, and knapsack selection are each well-understood techniques in isolation—but the recognition that code's nested structure demands a nested compression strategy. A function is both an indivisible unit for some purposes (you either include it or you don't—splitting it mid-body breaks its semantics) and a divisible unit for other purposes (within a relevant function, some blocks are boilerplate while others are critical). The two-stage design reflects this duality: coarse-grained compression treats functions as atomic and makes inclusion/exclusion decisions; fine-grained compression treats blocks as atomic and makes retention/pruning decisions within functions that survived the coarse stage.

The paper provides empirical evidence that both stages are necessary, but with a revealing asymmetry. The ablation study (Table VII) shows that removing the coarse-grained AMI ranking (replacing with similarity) hurts more (7.89 ES drop) than removing the entire fine-grained stage (1.45 ES drop). This doesn't mean the fine stage is unimportant—it means the coarse stage's decisions (which functions to include) are higher-stakes than the fine stage's decisions (which blocks within those functions to keep). Getting the right functions into the compressed context is the primary challenge; optimizing within those functions is a secondary refinement. This aligns with the paper's motivating observation in Figure 1: the catastrophic failure mode is missing Config entirely, not including Config but keeping one extra irrelevant method within it.

The adaptive budget allocation mechanism (Algorithm 1, Equation 5) represents a second subtle architectural insight: the importance scores from the coarse stage carry forward to inform fine-stage decisions. Functions with higher AMI receive proportionally more of the fine-grained budget (controlled by the $\beta$ parameter), creating a feedback loop where relevance at one granularity influences resource allocation at the next finer granularity. This is more sophisticated than a simple two-stage pipeline where each stage operates independently—it creates a form of hierarchical attention where global importance modulates local detail preservation. The ablation validates this: removing adaptive allocation (uniform budget across functions) costs 2.34 ES points, confirming that the importance signal from the coarse stage meaningfully improves fine-stage decisions.

A third architectural insight is the task-conditioned hyperparameter configuration. The same two-stage architecture with the same underlying mechanisms is deployed across three tasks by adjusting three parameters ($B$, $R_{\text{fine}}$, $\beta$), and these adjustments encode interpretable tradeoffs. Code completion needs tight budgets and modest fine pruning ($B=2$K, $R_{\text{fine}}=0.8$)—detail within retained functions matters. Code summarization needs generous coarse budgets and aggressive fine pruning ($B=5$K, $R_{\text{fine}}=0.3$)—broad structural awareness matters more than per-function detail. RepoQA disables fine pruning entirely ($R_{\text{fine}}=1.0$)—the output is a complete function, so intra-function pruning is counterproductive. These aren't tuned in a black-box grid search; they follow directly from task semantics, suggesting the architecture successfully disentangles the compression degrees of freedom into task-relevant and task-irrelevant axes.

This is an incremental but architecturally significant advance. The individual pieces (AMI scoring, knapsack selection, perplexity-based chunking) are adaptations of existing techniques to the code domain. The contribution is the synthesis—the recognition that code's hierarchical modularity (files contain classes, classes contain methods, methods contain blocks) maps naturally onto a hierarchical compression strategy, and that this mapping can be operationalized with off-the-shelf components arranged in a principled pipeline. For practitioners, this means LongCodeZip can be understood as a template for code compression that can be instantiated with different scoring functions, different segmentation methods, or different selection algorithms while preserving the two-stage structure.

Innovation 3: Perplexity-Based Block Boundary Detection as a Code-Aware Alternative to Syntactic Parsing

The fine-grained compression stage requires splitting functions into prunable units, and the paper introduces a method for doing this that is simultaneously more aligned with how LMs process code and simpler to implement than AST-based alternatives. Rather than using tree-sitter or language-specific parsers to identify statement boundaries, LongCodeZip uses the compression LM's own perplexity signal: compute per-line perplexity (without surrounding context), detect lines where perplexity spikes relative to neighbors, and treat those spikes as semantic boundaries.

This is a fundamentally different philosophy from prior code structure extractors. AST-based chunking (which cAST, a RAG-based baseline evaluated in Table VI, uses explicitly) answers the question: "According to the language grammar, where do syntactic units begin and end?" Perplexity-based chunking answers: "According to the language model's learned expectations, where do semantic units begin and end?" These are not the same. A for loop's header and body are syntactically distinct but semantically coupled—the model expects the body to follow the header, and its perplexity on the body tokens is low because the header constrains what follows. An AST would split them and potentially allow the knapsack to keep the header without the body, which would be syntactically valid but semantically incoherent. The perplexity-based approach would likely group them into a single block because the perplexity curve stays low across the boundary.

The significance extends beyond implementation convenience. This technique uses the same LM for segmentation that will later consume the compressed output, creating an implicit alignment between how the context is divided and how it will be read. The "semantic units" identified by perplexity boundaries are exactly the units that the generation LM expects as coherent chunks. This is a form of self-consistency in the compression pipeline: the compressor's understanding of code structure (expressed through perplexity) matches the generator's understanding (since they share pretraining), reducing the risk that the compressed context appears fragmented or incoherent to the downstream model.

The paper's ablation shows that perplexity-based chunking outperforms simple line-based chunking by 1.57 ES points (Table VII), and the knapsack-based block selection outperforms random line selection by 2.48 ES. These are modest but consistent improvements—the chunking method isn't the dominant contributor to overall performance (the AMI ranking is), but it provides a reliable signal within the fine stage.

This innovation is incremental but with transferable methodology. The core technique is adapted from Meta-Chunking (Zhao et al., 2024) for natural language, and the paper's contribution is demonstrating that it transfers to code with minimal adaptation. The broader implication is that code, despite its formal syntax, shares with natural language the property that semantic coherence manifests as perplexity smoothness—within a semantically unified region, the LM's uncertainty is low and relatively stable; at boundaries, it spikes. This suggests future work on code structure analysis (not just compression) could leverage LM perplexity as a signal complementary to, or replacing, syntactic parsing, particularly for tasks where the "units" are conceptual rather than grammatical.

Innovation 4: The Demonstration That Aggressive Context Compression Can Improve Performance—Not Just Reduce Cost

A striking and perhaps counterintuitive result appears across multiple experiments: LongCodeZip not only preserves performance relative to the no-compression baseline but sometimes exceeds it, particularly on complex reasoning tasks. On RepoQA with GPT-4o (Table V), LongCodeZip achieves 88.9% average accuracy at a 5.1× compression ratio, compared to 87.8% for the uncompressed baseline. On Claude-3.7-Sonnet, the same pattern appears: 88.9% with compression vs. 89.7% without—essentially tied. On the Long Code Completion task, Seed-Coder-8B achieves 63.11 ES at 5.6× compression vs. 64.04 ES without compression (Table II)—a gap of less than 1 point. Qwen2.5-Coder-7B actually sees a slight improvement: 57.55 ES with compression vs. 56.36 ES without.

This is not just "compression doesn't hurt"—it is a positive argument that removing irrelevant context improves model attention and reasoning. The mechanism is well-known from the "lost in the middle" literature (Liu et al., 2023, cited in Section I): LLMs struggle to attend to relevant information when it is embedded in long, mostly irrelevant contexts. By stripping away functions that have low AMI (i.e., functions that don't help predict the instruction), LongCodeZip effectively pre-filters the attention space, ensuring that the model's limited attention budget is focused on content that actually matters for the task. The compressed context may be shorter, but its information density is higher—every token that remains is there because the model itself (through the AMI signal) indicated it was useful.

This result has significant practical implications that go beyond efficiency. It suggests that the default approach of "include everything in the context window" is not only expensive but potentially harmful to output quality. For repository-level code tasks, where much of the codebase is infrastructural or unrelated to any specific query, removing irrelevant content is a form of noise reduction that directly improves the signal the generation model receives. This means context compression should be viewed not merely as a cost-saving measure (though it is that too) but as a quality-improving preprocessing step, analogous to how denoising or normalization improves downstream model performance in other domains.

The paper doesn't overclaim this finding—the improvements are not large and not universal (on code summarization with Qwen2.5-Coder-7B, the compressed score of 56.47 ties the baseline of 56.00; on DeepSeek-Coder-6.7B for summarization, compression outperforms the baseline 28.01 vs. 19.09, but the baseline performance is quite low and the task may be unusually sensitive to irrelevant context). But the consistency of the pattern—across models, tasks, and compression ratios—establishes a qualitative insight: context compression for code is a pareto improvement, not a tradeoff. You can get both lower cost and (at worst) equivalent quality, and in some settings higher quality. This moves the conversation from "how much quality are we willing to sacrifice for efficiency?" to "compression is a free lunch—we should do it by default."

This finding also supports the paper's architectural argument about the two-stage design. The reason LongCodeZip can sometimes beat the no-compression baseline is precisely that it does more than randomly or naively prune—it uses a semantically meaningful relevance signal (AMI) to make informed decisions about what to keep and what to discard. A random compressor would never exceed the baseline because it would discard critical and irrelevant information with equal probability. LongCodeZip's ability to beat the baseline is indirect evidence that its relevance judgments are accurate enough to selectively retain supporting information while discarding distracting information, achieving a net positive effect on the generation model's attention allocation.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. Three benchmarks are used: (1) Long Code Completion [1] — 500 Python examples filtered from the original test set to only include instances with context longer than 5,000 tokens (average context length 9,328.2 tokens, average ground truth length 12.4 tokens); (2) Long Module Summarization [21] — 139 examples filtered from the original 216 to include only instances exceeding 2,000 context tokens (average context length 10,809.6 tokens, average ground truth length 1,758.1 tokens, drawn from 43 Python repositories); (3) RepoQA [13] — 600 long-context code question-answering tests across 60 repositories in 6 programming languages (Python, Java, JavaScript, TypeScript, Rust, Go, C++), average context length 11,524.6 tokens, requiring the model to locate and reproduce a target function. The filtering on Long Code Completion and Long Module Summarization is deliberate: the authors want to evaluate on genuinely difficult long-context instances where compression is non-trivial, not on short prompts where the problem is easy regardless of method.

  • Base model(s). Three open-source code LLMs are evaluated: DeepSeek-Coder-6.7B [10], Qwen2.5-Coder-7B [11], and Seed-Coder-8B [12], all used in their instruct versions accessed via HuggingFace. Two closed-source models are also evaluated: GPT-4o and Claude-3.7-Sonnet. The model diversity is intentional: DeepSeek-Coder was released in 2023 and trained on data available only before March 2023 (which predates all benchmarks used, eliminating data leakage concerns — see Section VII); Qwen2.5-Coder was released in 2024 with a focus on long-context fidelity through multi-stage alignment; Seed-Coder was released in 2025 and represents the latest generation. This temporal spread tests whether LongCodeZip generalizes across training recipes and release epochs. The 6.7B–8B parameter range represents a practical deployment scale — large enough to be useful, small enough for the compression overhead to be tractable on a single A100-80G GPU.

  • Metrics. Three task-specific metrics are used. For code completion: Exact Match (EM) — the fraction of completions that exactly match the ground-truth solution — and Edit Similarity (ES) — the normalized edit distance between the generated and reference completions, following the LongCoder [1] evaluation protocol. For code summarization: CompScore, an LLM-as-judge metric where GPT-4o-mini acts as a referee, comparing the generated summary against the ground-truth summary after viewing both alongside the original code. To mitigate ordering bias, the referee evaluates both orderings (generated vs. reference, reference vs. generated), and CompScore is computed as CompScore = 0.5 × [P(sₒ ≻ ŝ) + (1 − P(ŝ ≻ sₒ))], where P(sₒ ≻ ŝ) is the probability the referee prefers the generated output sₒ over the reference ŝ. Scores range from 0 to 100, with 50 indicating equal preference. For RepoQA: retrieval accuracy, defined as the percentage of test questions where BLEU between the generated function and the target function exceeds 0.8, following the original RepoQA paper [13]. The primary efficiency metric across all tasks is compression ratio: Ratio = |C_original| / |C_compressed|, where the token counts are measured in the original tokenization of each model.

  • Baselines. Six categories of baselines are evaluated:

    • No Compression: The full original context is used without any modification, representing the upper performance bound.
    • No Context: Only the task instruction is provided with no code context at all, representing the lower bound.
    • Random Baselines: Random Token removes individual tokens uniformly at random; Random Line removes entire lines of code uniformly at random. Both serve as sanity checks — any method claiming to be "compressing intelligently" must substantially outperform random pruning.
    • Retrieval-based Methods: RAG (Sliding Window) chunks the code into fixed-size overlapping windows and retrieves the most similar chunks using UniXCoder-base [30] embeddings with cosine similarity. RAG (Function Chunking) splits code at function/class boundaries and retrieves using the same embedding model. These represent the dominant practical approach to context reduction for code LLMs and are the primary baselines LongCodeZip aims to outperform.
    • Code Compression Methods: DietCode [26] combines static frequency-based filtering with CodeBERT attention heuristics to prune low-impact tokens; it was originally implemented for Python and Java. SlimCode [27] applies rule-based token pruning using token types and program dependency graphs; it originally supported only Java, and the authors reproduced a Python version using tree-sitter for fair comparison. Both represent the prior state of the art in code-specific compression.
    • Text Compression Methods: LLMLingua [23] uses perplexity-based token importance scoring to prune uninformative tokens. LongLLMLingua [33] adds coarse-to-fine compression (document-level to token-level) with instruction-aware contrastive perplexity. LLMLingua-2 [34] trains a token classifier via data distillation from GPT-4 to identify essential vs. removable tokens. These represent the state of the art in prompt compression for natural language but were not designed for code.
    • Advanced RAG Methods on Code Completion: A3-CodGen [39], cAST [40], RepoGenix [41], and RLCoder [42] are evaluated on the Long Code Completion task in Table VI. These represent repository-level code completion systems that use sophisticated retrieval strategies. The paper explicitly notes these are complementary to LongCodeZip and could potentially be combined.

  • Generation budget / compute accounting. The paper measures "generation budget" differently from the search literature — there is no variable inference compute allocation. All baselines receive the same compressed context as input; the downstream LLM generates exactly one output (greedy or single-sample decoding). The relevant resource metric is compression ratio (how many tokens are removed) and compression time/memory (the overhead of running the compression pipeline itself). Total cost for a user is: compression cost (one-time, incurred by the compression model) + generation cost (incurred by the downstream LLM, proportional to compressed context length + generation length). The paper's efficiency analysis (Table IX) measures compression time (seconds of GPU compute), compression GPU memory (peak memory during compression), generation time (seconds of GPU compute for the downstream LLM), and generation GPU memory.

  • Cross-validation / statistical protocol. For the main comparison tables (II–V), results are averaged over 10 repeated experiments, and statistical significance is assessed via Wilcoxon signed-rank test comparing LongCodeZip against each baseline (reported as p < 0.001 in Section V-A). For hyperparameter tuning, the authors use "a small held-out set that did not overlap with the test data" (Section IV-E). The specific size and composition of this held-out set are not reported, which is a methodological weakness — readers cannot assess whether tuning was performed on a sufficiently representative sample or whether the hyperparameter choices might overfit to the tuning set. For the summarization task, the LLM-as-judge metric introduces its own variance; the paper mitigates ordering bias by averaging across both prompt orderings but does not report confidence intervals or referee model variance across repeated evaluations. For the FLOPs-matched comparison in Section 7 (if applicable), there is no explicit cross-validation described.

Main Quantitative Results

Code Completion (Table II, Figure 3)

The headline result on Long Code Completion is that LongCodeZip achieves ES scores that match or exceed the no-compression baseline at 4.3–5.6× compression ratios, while all other compression methods degrade performance substantially. With Qwen2.5-Coder-7B, LongCodeZip achieves 57.55 ES and 32.40 EM at a 4.3× compression ratio — actually slightly higher than the no-compression baseline (56.36 ES, 31.80 EM). With Seed-Coder-8B, LongCodeZip achieves 63.11 ES and 37.40 EM at 5.6× compression — a gap of only 0.93 ES and 2.80 EM below the no-compression baseline (64.04 ES, 40.20 EM). With DeepSeek-Coder-6.7B, LongCodeZip achieves 60.58 ES and 35.40 EM at 5.3× compression, exceeding the no-compression baseline (57.14 ES, 34.40 EM) by 3.44 ES and 1.00 EM points.

The comparison against the strongest baselines reveals the magnitude of LongCodeZip's advantage. On Qwen2.5-Coder-7B: RAG (Function Chunking) achieves 52.79 ES and 26.00 EM at a 3.1× compression ratio — LongCodeZip achieves 4.76 higher ES and 6.40 higher EM while compressing 28% more aggressively (4.3× vs. 3.1×). LongLLMLingua, the best text compression baseline, achieves only 23.88 ES and 9.00 EM at 3.2× compression on the same model — a catastrophic collapse of over 30 ES points from the no-compression baseline, strong evidence that token-level compression destroys code syntax in ways that render the context unusable for code LLMs. DietCode achieves 43.91 ES and 13.20 EM at 3.4×; SlimCode achieves 40.85 ES and 12.20 EM at 4.5×. Both are substantially below LongCodeZip despite comparable or lower compression ratios.

The performance-vs-compression curve (Figure 3) provides additional insight. LongCodeZip's ES score tracks the no-compression baseline across essentially the entire range of remaining context percentages, from 5% to 35%. At 5% remaining context (20× compression), LongCodeZip achieves approximately 57 ES, roughly matching the baseline — the compression is lossless in task-relevant information. In contrast, RAG-based methods (both sliding window and function chunking) show a roughly linear relationship between remaining context and ES, never approaching the baseline even at higher retention ratios. The random baselines (Random Token and Random Line) and LLMLingua-2 are essentially flat — adding more context doesn't help because the retained context is essentially random with respect to the task. This confirms that the AMI-based selection is not just "better than random" but genuinely identifies and preserves the critical subset of the context that carries all the information needed for successful completion.

Code Summarization (Table III)

On Long Module Summarization, LongCodeZip again consistently outperforms all compression baselines, though the absolute CompScore values are lower across the board (reflecting that summarization is a fundamentally harder task with longer, more open-ended outputs than completion). With DeepSeek-Coder-6.7B, LongCodeZip achieves 28.01 CompScore at 2.5× compression — a substantial improvement over the no-compression baseline of 19.09. This is one of the clearest examples of compression improving performance: the uncompressed context contains so much irrelevant code that it actively degrades summarization quality, and removing that noise helps the model focus. With Qwen2.5-Coder-7B, LongCodeZip achieves 56.47 CompScore at 1.7× — essentially tied with the no-compression baseline of 56.00. With Seed-Coder-8B, LongCodeZip achieves 55.07 at 3.5×, exceeding the baseline of 44.95 by over 10 CompScore points.

The comparison against baselines on summarization is particularly informative because RAG-based methods do not show the same advantage they had on code completion. RAG (Sliding Window) achieves 53.50 on Qwen2.5-Coder-7B — competitive with LongCodeZip's 56.47, but at a lower compression ratio (1.7× vs. LongCodeZip's 1.7× at a comparable setting). On DeepSeek-Coder-6.7B, RAG (Sliding Window) achieves 22.95, below LongCodeZip's 28.01. LLMLingua-2 achieves 52.99 on Qwen2.5-Coder-7B — quite competitive — but the paper notes that the average context lengths for summarization (10,809.6 tokens) are longer than for completion, and the achieved compression ratios (1.7–3.5×) are lower, suggesting less headroom for aggressive pruning. The task inherently requires more context to be preserved.

A notable outlier: LongLLMLingua achieves 49.73 on Seed-Coder-8B at 2.4× compression, vs. LongCodeZip's 55.07 at 3.5×. The 5.34 point gap is smaller than on other tasks, suggesting that LongLLMLingua's coarse-to-fine document-level approach partially mitigates the code-structure-destroying problems of pure token-level compression, but still falls short of code-aware methods.

Question Answering / Retrieval (Table IV)

On RepoQA, where the task is essentially "find and reproduce the exact function," LongCodeZip achieves its most dramatic relative improvements. With Qwen2.5-Coder-7B, LongCodeZip achieves 87.2% average accuracy at 4.5× compression — slightly exceeding the no-compression baseline of 86.0%. With Seed-Coder-8B, LongCodeZip achieves 80.7% at 5.3×, vs. 69.0% without compression — an 11.7 percentage point absolute improvement. With DeepSeek-Coder-6.7B, LongCodeZip achieves 75.3% at 5.3×, vs. 38.3% without compression — nearly doubling the accuracy. The DeepSeek-Coder-6.7B baseline is notably low (38.3%), suggesting that this earlier model is particularly susceptible to the "lost in the middle" problem on this benchmark, and compression that removes irrelevant functions provides outsized benefits.

The per-language breakdown in Table IV reveals consistent patterns. On Qwen2.5-Coder-7B, LongCodeZip exceeds the no-compression baseline on Python (92 vs. 84), C++ (78 vs. 77 — tied), Java (87 vs. 89 — slightly below), Rust (86 vs. 83), Go (95 vs. 90), and TypeScript (85 vs. 93 — below). The method performs well across all six languages without language-specific tuning, confirming the model-agnostic and language-agnostic claims. The compression ratios (4.5–5.3×) are higher than for summarization, consistent with the task demanding less broad context.

The catastrophic failure of text compression methods on RepoQA is striking and informative. On Qwen2.5-Coder-7B: LLMLingua achieves 8.7% average accuracy (vs. 86.0% no-compression); LLMLingua-2 achieves 2.8% (vs. 86.0%). These are not marginal degradations — they represent essentially complete loss of the ability to identify the target function. The explanation (discussed qualitatively in Section V-A) is that token-level pruning destroys the function's syntactic structure, making it impossible for the downstream LLM to extract a complete, correct function. LongLLMLingua does better (71.3% at 4.3×) due to its coarse-to-fine strategy that preserves document-level structure, but still trails LongCodeZip (87.2%) substantially. RAG (Sliding Window) achieves 67.5% at 3.7×; RAG (Function Chunking) achieves only 54.3% at 4.3×. The RAG failure on RepoQA aligns with the paper's motivating argument: retrieval by embedding similarity misses functions that are functionally coupled to the query but lexically dissimilar.

DietCode and SlimCode are evaluated only on Python, Java, and the languages for which they were originally implemented (explaining the dashes in Table IV for C++, TypeScript, Rust, Go). On Qwen2.5-Coder-7B, DietCode achieves 26.0% average (17.0 on Python, 35.0 on Java); SlimCode achieves 34.0% average (20.0 on Python, 48.0 on Java). Both are substantially below LongCodeZip, confirming that prior code compressors designed for single-function contexts do not transfer to the long-context, multi-file setting.

Performance with Closed-Source Models (Table V)

LongCodeZip's effectiveness transfers to commercial API models, which is critical for practical deployment since cost savings on these models are the primary economic motivation. On GPT-4o: LongCodeZip achieves 64.72 ES and 38.80 EM at 4.3× compression on Long Code Completion, compared to 65.13 ES and 40.80 EM without compression — a gap of only 0.41 ES and 2.00 EM. On Long Module Summarization, LongCodeZip achieves 59.04 CompScore vs. 58.42 without compression (slight improvement). On RepoQA, LongCodeZip achieves 88.9% average accuracy vs. 87.8% without compression — again a slight improvement. On Claude-3.7-Sonnet: LongCodeZip achieves 66.27 ES and 40.20 EM at 4.3× vs. 66.24 ES and 41.20 EM without compression — essentially identical performance. On RepoQA, Claude-3.7-Sonnet with LongCodeZip achieves 88.9% vs. 89.7% without compression (near tie). On summarization, LongCodeZip achieves 61.47 CompScore vs. 60.72 without compression (improvement).

Two patterns are notable. First, the compression ratios are consistent across open-source and closed-source models (4.3× for completion, 1.7× for summarization, 5.1× for RepoQA), confirming that the hyperparameters tuned on open-source models transfer. Second, on RepoQA specifically, LongCodeZip slightly exceeds the no-compression baseline for both closed-source models — removing irrelevant context actually helps these powerful models locate the target function, even though they have long context windows and sophisticated attention mechanisms. This supports the argument that compression is not just a cost hack but a genuine quality improvement for retrieval-like code tasks.

Comparison with Advanced RAG Methods (Table VI)

Table VI evaluates LongCodeZip against four recent RAG-based repository-level code completion systems on Seed-Coder-8B and Claude-3.7-Sonnet. The results are consistent: LongCodeZip outperforms all four RAG methods at higher compression ratios. On Seed-Coder-8B: A3-CodGen achieves 58.70 ES and 33.10 EM at 3.8×; cAST achieves 57.35 ES and 30.90 EM at 4.1×; RepoGenix achieves 60.28 ES and 34.70 EM at 3.5×; RLCoder achieves 58.14 ES and 32.30 EM at 4.0×. LongCodeZip achieves 63.11 ES and 37.40 EM at 5.6× — 2.83 ES and 2.70 EM higher than the best RAG method (RepoGenix) while compressing 60% more aggressively. On Claude-3.7-Sonnet, the same pattern holds: LongCodeZip achieves 66.27 ES at 4.3× vs. 62.76 for RLCoder at 4.0× (the best RAG method on this model). The paper frames this not as "RAG is bad" but as "compression and retrieval are complementary" — the RAG methods retrieve candidate contexts, and LongCodeZip could compress those retrieved contexts before feeding to the LLM, potentially combining the strengths of both approaches. This combined pipeline is not evaluated.

Efficiency Analysis (Table IX)

Table IX reports wall-clock time and GPU memory measurements for compression and generation on the Long Code Completion task with Qwen2.5-Coder-7B. Compression overhead: LongCodeZip requires 2.58 seconds of compression time and 0.69 GB of additional GPU memory. This is higher than the retrieval-based RAG (0.53 seconds, 1.07 GB) but substantially lower than DietCode (15.23 seconds, but 0.0 GB since it's CPU-based). LLMLingua-2 compresses in 0.65 seconds but uses 4.71 GB of GPU memory — significantly more memory-hungry than LongCodeZip despite being token-level. SlimCode is the fastest (0.35 seconds, 0.0 GB) but produces poor downstream performance.

Generation savings: The generation time drops from 15.70 seconds (no compression) to 6.59 seconds (LongCodeZip at 4.3× compression) — a 58% reduction. The generation GPU memory drops from Base + 3.48 GB to Base + 0.81 GB — a 77% reduction. The net effect: despite 2.58 seconds of compression overhead, the total pipeline time (compression + generation) is 9.17 seconds, vs. 15.70 seconds for no compression — a 42% reduction in end-to-end latency. If the compression model is swapped for a 0.5B model (which Table VIII shows achieves comparable ES scores), the compression time would drop further, making the net latency savings even larger.

Cost implications: The paper notes that for commercial APIs where pricing is per input token, a 4.3× compression directly translates to a ~77% reduction in input token costs, which for models like GPT-4o (priced at dollars per million tokens) can mean savings of cents to dollars per query. For high-volume deployment (CI/CD pipelines, IDE integrations processing thousands of queries daily), these per-query savings compound substantially.

Cross-Model Generalization (Table VIII)

Table VIII presents a comprehensive transferability matrix showing ES scores on Long Code Completion when the compression model differs from the generation model. The rows are compression models (DeepSeek-Coder-6.7B, Seed-Coder-8B, Qwen2.5-Coder-7B plus smaller Qwen2.5-Coder variants at 0.5B, 1.5B, and 3B); the columns are generation models (the three 6.7B–8B models). The key finding: the average ES across all generation models is remarkably stable regardless of which model is used for compression. The average ES ranges from 59.54 (using DeepSeek-Coder-6.7B as compressor) to 60.58 (using Qwen2.5-Coder-7B as compressor) — a spread of only 1.04 points. The 0.5B Qwen2.5-Coder achieves an average ES of 60.13 — better than the 7B DeepSeek-Coder as compressor (59.54) and within 0.45 points of using the largest compression model (60.58).

This is a genuinely striking result. A model with 0.5B parameters — roughly 14× smaller than the generation models — produces AMI rankings that are essentially equivalent to those from models 14× larger. This strongly supports the paper's claim that the relevance knowledge captured by AMI is a universal property of code pretraining that transfers across model scales and architectures. For practical deployment, this means the compression overhead can be reduced dramatically (a 0.5B model is fast and fits in a few hundred MB of GPU memory) without sacrificing compression quality. The paper doesn't provide separate efficiency numbers for the 0.5B compression model, but the implications are clear from the trend: faster, cheaper compression with no downstream quality penalty.

Ablation Studies and Robustness Checks

All ablations are performed on the Long Code Completion task with Qwen2.5-Coder-7B (Table VII). The full LongCodeZip configuration achieves 57.55 ES and 32.40 EM at 4.3× compression.

  • Coarse-grained ranking replaced with similarity-based ranking: The AMI-based function ranking is replaced with UniXCoder embedding similarity ranking (cosine similarity between instruction embedding and function embedding). ES drops from 57.55 to 49.66 (a decrease of 7.89), EM drops from 32.40 to 25.20 (a decrease of 7.20). This is the single largest ablation impact in the entire study, confirming that the choice of relevance metric — mutual information vs. lexical similarity — is the dominant design decision. The ratio remains 4.3×, so the degradation is not due to different compression ratios but to genuinely worse selection of which functions to retain.

  • Coarse-grained ranking replaced with random ranking: Functions are ranked randomly. ES drops to 39.76 (decrease of 17.79), EM drops to 11.50 (decrease of 20.90). This is essentially catastrophic — performance collapses to near the No Context baseline (38.14 ES, 9.60 EM). This establishes the lower bound and confirms that AMI-based ranking is extracting a meaningful signal, not just exploiting some trivial property of the context structure.

  • w/o Fine-grained Compression: The entire fine-grained stage is removed; only coarse-grained function selection is applied with the same final token budget. ES drops to 56.10 (decrease of 1.45), EM drops to 31.20 (decrease of 1.20), ratio 4.2×. The relatively modest drop confirms that the coarse stage does the heavy lifting — function-level selection accounts for most of the compression benefit — but the fine stage provides a measurable incremental improvement. This is consistent with the architecture rationale: getting the right functions into the compressed context is the primary challenge; optimizing within those functions is secondary but still valuable.

  • w/o Adaptive Budget Allocation: The adaptive budget allocation (Algorithm 1) is removed; all retained functions receive the same uniform retention ratio R_base. ES drops to 55.21 (decrease of 2.34), EM drops to 29.40 (decrease of 3.00), ratio 4.3×. The 2.34 ES drop is larger than removing the entire fine-grained stage, which initially seems paradoxical — how can removing a sub-component hurt more than removing the whole stage? The resolution: when fine-grained compression is entirely removed, the coarse stage fills the full budget with complete functions, so no intra-function pruning occurs, and adaptive allocation is moot. When fine-grained compression is active but uniform (no adaptive allocation), the budget is distributed suboptimally — important functions are under-compressed (wasting tokens on less relevant internal blocks) while less important functions are over-compressed (losing critical detail). The adaptive allocation corrects this misallocation, so removing it hurts. The interaction between the two components is non-additive, which the paper doesn't explicitly discuss but which the numbers make clear.

  • Perplexity-based chunking replaced with line-based chunking: Perplexity-based block boundary detection is replaced with simple line-by-line splitting (each line is its own block). ES drops to 55.98 (decrease of 1.57), EM drops to 31.20 (decrease of 1.20), ratio 4.3×. The perplexity-based segmentation provides a modest but consistent improvement over naive line splitting. This aligns with the qualitative analysis in Figure 4: perplexity boundaries group semantically related lines (e.g., a multi-line if statement) into coherent blocks that the knapsack can sensibly keep or remove as units, whereas line splitting would fragment these logical units and potentially produce syntactically broken selections.

  • Knapsack-based block selection replaced with random line selection: Within each function, blocks are selected randomly rather than via the knapsack DP. ES drops to 55.07 (decrease of 2.48), EM drops to 29.00 (decrease of 3.40), ratio 4.3×. The 2.48 ES improvement from the knapsack over random selection confirms that the AMI-based value scores for blocks (whether computed per-block or inherited from parent functions) carry meaningful signal about which blocks are worth keeping. If all blocks had identical value, random and knapsack selection would perform equivalently; the fact that they differ confirms that AMI varies meaningfully across blocks within a function.

Critical Assessment

Claim 1: "LongCodeZip achieves up to 5.6× compression without sacrificing performance"

The claim is well-supported for code completion and RepoQA, but with task-dependent nuance. On code completion with Seed-Coder-8B, LongCodeZip achieves 63.11 ES at 5.6× vs. 64.04 ES no-compression — a 0.93 ES gap that is arguably "without sacrificing performance" (Table II). On Qwen2.5-Coder-7B, the compressed ES (57.55) slightly exceeds the baseline (56.36). On DeepSeek-Coder-6.7B, the compressed ES (60.58) meaningfully exceeds the baseline (57.14). On GPT-4o and Claude-3.7-Sonnet (Table V), the gaps are 0.41 and 0.03 ES respectively — genuinely negligible. On RepoQA, the claim holds more strongly: LongCodeZip usually exceeds the no-compression baseline on this task (Table IV, Table V), particularly on weaker generation models (DeepSeek-Coder-6.7B: 75.3% vs. 38.3%; Seed-Coder-8B: 80.7% vs. 69.0%).

However, the claim does not hold uniformly for code summarization (Table III). On Qwen2.5-Coder-7B, the compressed CompScore (56.47) ties the baseline (56.00) — "without sacrificing" is true but the compression ratio is only 1.7×, far from 5.6×. On DeepSeek-Coder-6.7B, compression improves performance (28.01 vs. 19.09), but at only 2.5× compression. On Seed-Coder-8B, compression improves performance (55.07 vs. 44.95) at 3.5×. The "up to 5.6×" figure is achievable only on tasks with specific characteristics (completion and RepoQA); on summarization, the maximum lossless compression ratio is lower (~1.7–3.5×). This is not a weakness of the method — it reflects that summarization genuinely needs more context — but the abstract's universal "up to 5.6×" framing overstates the typical compression ratio across tasks.

More fundamentally, the "without sacrificing performance" framing is measured against a greedy/single-sample decoding baseline. If the no-compression baseline used best-of-N or majority voting (as is common in production LLM deployments), the baseline performance would be higher, and the compressed-vs-uncompressed gap might widen. The paper does not explore whether compression interacts differently with more sophisticated decoding strategies. A compressed context might, for instance, reduce the diversity of generated outputs in a way that hurts majority-voting performance even when single-sample performance is preserved.

Claim 2: "LongCodeZip consistently outperforms all compression baselines"

Strongly supported. Across all three tasks, all models, and all compression ratios evaluated, LongCodeZip achieves the highest task performance among compression methods. The margins are substantial: +4.76 ES over RAG (Function Chunking) on Qwen2.5-Coder-7B for code completion; +17.2 percentage points over RAG (Sliding Window) on RepoQA with the same model; +10+ CompScore over most baselines on summarization. The Wilcoxon signed-rank test (p < 0.001 across 10 repeated experiments) provides statistical rigor, though the paper doesn't report confidence intervals for the performance gaps.

A caveat: LLMLingua-2 achieves 52.99 CompScore on Qwen2.5-Coder-7B for summarization (Table III), compared to LongCodeZip's 56.47. This is a competitive result and the gap (3.48) is smaller than on other tasks. If summarization were the only target application, the case for LongCodeZip over LLMLingua-2 would be weaker, especially given that LLMLingua-2 is a simpler, single-stage token classifier. LongCodeZip's advantage is most decisive on tasks where code structure matters most (completion, RepoQA) and less dramatic on summarization where broader semantic understanding may be more forgiving of structural disruption.

A more significant caveat: the paper does not compare against a retrieval + compression hybrid baseline. The authors state that RAG methods are "complementary to our compression approach and could potentially be combined" (Section V.A), but never demonstrate this combination. The natural comparison would be: RAG retrieves candidate functions by embedding similarity, then LongCodeZip compresses those retrieved candidates using AMI-based selection. This could potentially outperform either method alone, and without this experiment, the claim that LongCodeZip "outperforms" retrieval-based methods tells us about the compression step in isolation but not about whether a combined system would be better still.

Claim 3: "LongCodeZip is model-agnostic and exhibits strong cross-model generalization"

Strongly supported by Table VIII. The transferability matrix shows that across all compression-model/generation-model pairs, the average ES varies by only ~1 point. Using a 0.5B model for compression and a 7–8B model for generation yields essentially the same downstream performance as using the generation model itself for compression. This is the most robust finding in the paper — it's a 6 × 3 matrix of experiments, covers three model families released across a two-year span, and produces consistent results without any evident outliers or interactions.

However, there is an important scope limitation: all models tested are decoder-only code LLMs trained on similar corpora (GitHub code, primarily). The "model-agnostic" claim would be more convincing if it included an encoder-decoder model (CodeT5+), a model trained primarily on natural language with code as a secondary modality (GPT-4, though Table V partially addresses this by evaluating GPT-4o and Claude as generation models), or a model with a fundamentally different pretraining objective. The transferability across code LLMs is impressive, but "model-agnostic" might overstate the finding for truly out-of-family architectures.

Additionally, the compression model always computes AMI using the instruction as the target and context functions as the conditioning signal. For a model that was not trained on code, the AMI scores would presumably degrade — the model wouldn't "know" that Config and train_model functionally couple — and the compression quality would drop. The cross-model transferability works because the models share code pretraining knowledge, not because AMI is universally architecture-independent.

Claim 4: "The two-stage architecture is necessary and each component contributes"

Supported with an important nuance about interaction effects. The ablation study (Table VII) demonstrates that each component removal degrades performance, but the relative magnitudes reveal an asymmetric architecture: the coarse-grained ranking (AMI vs. similarity) is by far the dominant component (7.89 ES drop), while the fine-grained components collectively contribute 1.45–2.48 ES each. This doesn't mean the fine stage is unnecessary — 1.45 ES is a meaningful improvement — but it does mean that a practitioner with extremely tight latency constraints could drop the fine stage and retain ~97% of the performance at a slightly lower compression ratio (4.2× vs. 4.3×). The paper doesn't explore this tradeoff explicitly for practitioners.

An interaction effect the paper doesn't discuss: removing adaptive budget allocation (2.34 ES drop) hurts more than removing the entire fine stage. As noted above, this is because the "remove fine stage" ablation keeps functions intact and fills the budget with complete functions, sidestepping the allocation problem entirely. The adaptive allocation is valuable precisely because intra-function pruning is being performed; without it, pruning is applied uniformly and suboptimally. This suggests the ablation on removing the fine stage should be interpreted carefully — it's not that the fine stage is less important than the coarse stage (it is), but that the comparison between "uniform fine pruning" and "no fine pruning" conflates the decision to prune with the decision about how to allocate pruning. A cleaner ablation would be: no fine pruning (coarse only) vs. adaptive fine pruning vs. uniform fine pruning on a fixed budget, but the paper reports all three.

Missing Experiments That Would Strengthen the Paper

  • Sensitivity to hyperparameters. The paper sets B, R_fine, and β based on a held-out set but never reports how sensitive performance is to these choices. Would a 20% change in B cause a 5% or 20% drop in ES? This matters for practitioners tuning on new tasks. The β = 0.5 setting is used across all three tasks without variation — is 0.5 actually optimal for any of them, or is β simply not very important?

  • AMI computation cost breakdown. The efficiency analysis reports total compression time (2.58s) but doesn't break down how much of this is AMI scoring for function ranking vs. block boundary detection vs. knapsack DP. This matters because the cross-model results suggest a 0.5B model would be much faster, but without the breakdown, readers can't estimate how much faster or identify which part of the pipeline is the bottleneck.

  • Comparison with a "just use the top-K functions" baseline. The paper's coarse stage selects functions by AMI ranking and then applies fine-grained pruning within them. A simpler baseline would be: rank functions by AMI, take the top K functions (without fine pruning), fit to budget. The "w/o Fine-grained Compression" ablation approximates this, but it uses the same hyperparameters — a direct comparison on a wider range of budgets would clarify whether the fine stage is worth its complexity for typical use cases.

  • Effect of compression on generation diversity. All evaluations use single-sample or greedy decoding. If the generation model were run with temperature > 0 and multiple samples drawn, would the compressed context produce less diverse outputs? This matters for tasks where diversity is valuable (e.g., generating multiple candidate completions for a developer to choose from) and the paper provides no data.

  • Instruction ambiguity stress test. The paper notes (Section VI.B) that "when the context either lacks information relevant to the task instruction or when it is difficult to align an ambiguous instruction with any segment of the context, our method may struggle." This is anecdotally reported but never systematically tested. Constructing a benchmark where instructions are deliberately underspecified or where the context deliberately lacks the needed information would characterize this boundary condition quantitatively.

6. Limitations and Trade-offs

The Cost of AMI-Based Difficulty Estimation Is Not Accounted for in the Headline Efficiency Gains

The assumption or constraint: The entire two-stage compression pipeline depends on computing approximated mutual information (AMI) scores between the task instruction and each candidate function chunk. This requires one forward pass of the compression LM per function—for a context containing 100 functions, that means approximately 100 forward passes, each processing the function plus the instruction tokens. The paper's efficiency analysis (Table IX) reports a total compression time of 2.58 seconds for Qwen2.5-Coder-7B on the Long Code Completion task, but this cost is reported as a monolithic overhead and is not amortized against the headline claim of "up to 5.6× compression ratio without sacrificing task performance" (Abstract, Section V-A). The paper acknowledges (Section III-C) that the coarse-grained stage uses "a configurable fine-grained compression ratio $R_{\text{fine}}$" to control how many functions pass to the fine stage, and the AMI computation is the dominant cost in this pipeline.

The consequence: The reported compression ratios and generation time savings (e.g., "reduces generation time from 15.7s to 6.6s" in Section V-D) are measured excluding the cost of computing AMI scores for all functions in the uncompressed context. This creates an apples-to-oranges comparison against baselines like RAG, which has substantially lower compression overhead (0.53 seconds, Table IX) or SlimCode (0.35 seconds). For a single query, the 2.58-second compression overhead plus 6.59-second generation time (9.17 seconds total) is still faster than 15.70 seconds for no compression—but the gap narrows from 9.1 seconds saved to 6.5 seconds saved. For high-throughput scenarios where the same context is reused across many queries (e.g., a developer asking multiple questions about the same file), the AMI computation can be amortized, but the paper does not characterize this regime. More critically, if the number of functions in the context grows (e.g., in very large repositories), the AMI computation cost scales linearly with the number of functions while the uncompressed context length also grows, potentially eroding the net latency benefit for extremely large codebases.

What evidence exists in the paper: Table IX reports the 2.58-second compression time for the code completion task with Qwen2.5-Coder-7B, but this is a single measurement at a specific context size (~9,328 tokens on average, Table I) and function count (not reported). The paper does not provide a scaling analysis of compression time as a function of context length or number of functions. The cross-model experiments (Table VIII) demonstrate that a 0.5B compression model achieves comparable AMI ranking quality—the paper notes this "will significantly reduce compression time and memory overhead" (Section V-C)—but provides no quantitative measurements of the speedup. The compression GPU memory overhead (Base + 0.69 GB, Table IX) is reported for the 7B compression model, but the memory scaling for smaller models is not characterized. Without these measurements, a practitioner cannot estimate the compression cost for their specific context sizes or determine the break-even point where compression overhead exceeds generation savings.

Mitigation status: The paper partially acknowledges this issue by noting in Section V-C that "using such small models will significantly reduce compression time and memory overhead, making our approach particularly suitable for resource-constrained scenarios." However, this is a qualitative suggestion rather than a measured claim. The paper does not report compression time, memory, or downstream performance for the 0.5B compression model in an efficiency table comparable to Table IX. The implication that a smaller model would be faster is reasonable (a 0.5B model has roughly 1/14 the parameters of the 7B model, and forward passes scale roughly linearly with parameter count), but the actual speedup depends on implementation details (batch size, memory bandwidth, KV-cache management when computing AMI for multiple functions) that are not discussed. The authors do not suggest a future direction for reducing the AMI computation cost, such as function batching, speculative early termination of AMI computation for clearly irrelevant functions, or training a lightweight AMI predictor.

The Method Fundamentally Cannot Handle Contexts That Lack Relevant Information

The assumption or constraint: LongCodeZip operates on the principle of selecting and preserving the most relevant parts of the given context. It does not retrieve new information from outside the provided context, nor does it augment the context with external knowledge. The authors explicitly acknowledge this fundamental bound in Section VI-B:

"In particular, when the context either lacks information relevant to the task instruction or when it is difficult to align an ambiguous instruction with any segment of the context, our method may struggle to identify and preserve useful blocks."

This is not a bug in the implementation—it is an inherent architectural limitation of any compression-only approach. If the long code context provided to LongCodeZip does not contain the answer (e.g., the repository is missing a critical dependency, the instruction refers to a library not present in the context, or the needed function is in a file that wasn't included), then no amount of intelligent selection can create the missing information. The best LongCodeZip can do in this scenario is produce a compressed context that still lacks the needed information—it cannot flag that the context is insufficient or escalate to a retrieval system.

The consequence: This limitation means LongCodeZip is not a substitute for retrieval but rather a complement to it, and using LongCodeZip without a preceding retrieval step may silently fail on queries where the provided context is incomplete. In the paper's experimental setup, the "long context" for each task is constructed to contain the necessary information (the benchmarks are designed this way), so this failure mode does not manifest in the reported results. However, in real-world deployment scenarios—where a developer might ask a question about a repository without knowing whether the relevant file was included in the context window—LongCodeZip would compress what it has without any signal that the compression is discarding context that is actually necessary (none of it is) or that the compressed output is still insufficient (because the needed information was never present).

A related but subtler consequence: even when the context does contain relevant information, if the task instruction is ambiguous or poorly aligned with any specific code segment, the AMI scores may be uniformly low or noisy. Functions that would be highly relevant if the instruction were more specific will receive low AMI scores because the model cannot establish the connection from the vague instruction. For example, an instruction like "fix the bug" with no additional specification would produce AMI scores that are essentially random with respect to the actual bug location. LongCodeZip would still select functions (greedy selection under budget takes whatever scores highest), but the selection would be arbitrary rather than informed.

What evidence exists in the paper: The paper provides no systematic evaluation of this failure mode. The anecdotal mention in Section VI-B is the only acknowledgment. No experiment constructs contexts that deliberately lack relevant information and measures whether LongCodeZip's downstream performance degrades gracefully (e.g., to the No Context baseline) or catastrophically (e.g., to worse than random, because the model receives misleading partial context). No experiment varies instruction specificity (from highly specific "complete function X with signature Y" to deliberately ambiguous "improve this code") to measure the sensitivity of AMI ranking to instruction quality. The paper's datasets use well-specified, task-focused instructions (code completion with explicit function stubs, summarization with clear module targets, RepoQA with natural language function descriptions), which avoids exposing this limitation but also leaves its severity uncharacterized.

Mitigation status: The paper does not attempt to mitigate this limitation. It is acknowledged as a failure mode in Section VI-B but is not addressed through any mechanism (e.g., a confidence score that flags low-AMI contexts for human review, or a fallback that routes ambiguous instructions to a retrieval system). The authors' framing in Section VIII-B—"Unlike these approaches that are specifically designed for repository-level code completion, we propose a training-free code context compression technique that provides broader applicability"—positions LongCodeZip as complementary to retrieval, which implicitly acknowledges that retrieval (to ensure the context contains relevant information) should precede compression. However, this complementarity is stated as a possibility ("could potentially be combined") rather than evaluated or designed for, leaving the integration burden entirely on the practitioner.

The Evaluation Is Limited to a Single Domain (Code Benchmarks) with Moderate-Scale Models, Leaving Generalization to Other Code Tasks, Languages, and Model Scales Unverified

The assumption or constraint: All experiments are conducted on three code-specific benchmarks: Long Code Completion (Python only), Long Module Summarization (Python only), and RepoQA (6 languages but all retrieval-style QA). The evaluated generation models span 6.7B–8B parameters (open-source) plus two commercial API models of unknown size. The paper claims in Section VIII-B that "to the best of our knowledge, our approach is the first to explicitly target long-context compression in code LLMs, providing a training-free and model-agnostic solution," and in Section VII acknowledges as a threat to validity that "our findings may be specific to the datasets, programming languages, or LLMs used."

The consequence: Several important generalization questions are left unanswered. First, task diversity: the three tasks (completion, summarization, retrieval) cover important code intelligence applications, but do they represent the full range of long-context code scenarios? Notably absent are: repository-level code translation (translating an entire project across languages while preserving behavior), code review generation (producing a review of a large pull request), multi-file refactoring (suggesting changes across a codebase), and test generation (writing tests for a module given its implementation and dependencies). Each of these tasks has different information requirements—refactoring might need broad structural awareness (favoring summarization-like configurations), while test generation might need focused local detail (favoring completion-like configurations). The paper's three task-specific hyperparameter configurations (Section IV-E) demonstrate that task sensitivity exists, but without evaluating on more diverse tasks, a practitioner cannot determine whether the existing configurations cover their use case or whether new tuning would be required.

Second, model scale. The open-source models tested (6.7B–8B parameters) represent a practical but narrow range. It is unknown whether LongCodeZip's benefits would be larger on smaller models (which struggle more with long contexts) or smaller on much larger models (which may handle long contexts better natively). The commercial models (GPT-4o, Claude-3.7-Sonnet) partially address this by representing larger, more capable systems, but their parameter counts are unknown, and the API abstraction prevents measuring whether the AMI computation (which requires token log-probabilities, potentially not exposed by all commercial APIs) would work with models accessed only through chat endpoints. The cross-model experiments (Table VIII) are limited to three model families (DeepSeek-Coder, Qwen2.5-Coder, Seed-Coder), all of which are decoder-only code-specialized LLMs. The claim of "model-agnostic" operation has not been tested with encoder-decoder architectures (CodeT5+), general-purpose LLMs not specifically fine-tuned for code, or very large models (70B+ parameters) where the cost-benefit tradeoff of compression may shift.

Third, language coverage. Long Code Completion and Module Summarization are Python-only. RepoQA covers six languages, but Table IV reveals substantial per-language variation in both baseline performance and LongCodeZip's improvement. On DeepSeek-Coder-6.7B, LongCodeZip improves Python accuracy from 21.0% to 76.0% (a 55-point gain), but Go accuracy only from 59.0% to 79.0% (a 20-point gain). On Qwen2.5-Coder-7B, TypeScript actually degrades from 93.0% to 85.0% with compression. These language-specific differences are not analyzed or explained. It is plausible that languages with more verbose syntax (Java, C++) benefit more from compression than languages with concise syntax (Python, Go), or that the perplexity-based block boundary detection works differently across languages with different structural conventions (Python's significant whitespace vs. C-style brace-delimited blocks). Without cross-language analysis on the other tasks, the generalizability of compression ratios and performance preservation across the full spectrum of programming languages used in practice (Ruby, PHP, Swift, Kotlin, etc.) is uncertain.

What evidence exists in the paper: The paper acknowledges these limitations explicitly in Section VII (Threats to Validity): "Our findings may be specific to the datasets, programming languages, or LLMs used. To improve generalizability, we evaluated our approach across diverse datasets, languages, model families, and in cross-model settings." The authors present the multi-benchmark, multi-model, multi-language results as evidence of generalizability. However, "diverse datasets" means three benchmarks, all from the code domain, with similar long-context construction methodologies. "Diverse languages" means six languages on only one task (RepoQA), with the other two tasks being Python-only. "Diverse model families" means three code-specialized LLM families with similar architectures. This is stronger evidence than a single-benchmark, single-model evaluation would provide, but falls short of establishing the broad generalizability the paper's framing implies.

Mitigation status: The paper presents the multi-benchmark, multi-model, multi-language evaluation as the primary mitigation, which is reasonable given the scope of a single paper. The cross-model results (Table VIII) are genuinely convincing for transferability across the tested model family range. The RepoQA six-language results demonstrate that the method works across languages without language-specific configuration. However, the authors do not propose specific future experiments to address the remaining generalization gaps (additional tasks, larger models, additional language families). The threat-to-validity discussion in Section VII is appropriately honest but does not provide a roadmap for the community to close these gaps.

The Perplexity-Based Block Segmentation Method Lacks Rigorous Validation as a Code Structure Detection Mechanism

The assumption or constraint: The fine-grained compression stage relies on perplexity-based block boundary detection to segment functions into semantically coherent blocks. This method—adapted from natural language work (Zhao et al., 2024, Meta-Chunking)—marks a line as a block boundary when its unconditional perplexity exceeds that of the previous line by more than $\alpha$ times the standard deviation of perplexity across all lines in the function (Section III-D). The paper provides a qualitative case study in Figure 4 illustrating that detected boundaries align with some structural transitions, but provides no quantitative evaluation of how well these boundaries correspond to actual code structure—as measured by AST node boundaries, control flow boundaries, data dependency boundaries, or human-labeled semantic segmentation.

The sensitivity parameter $\alpha$ is mentioned in the text but its value is never specified—not in the main text, not in the hyperparameter configuration section (Section IV-E), and not in any appendix. The standard deviation $\sigma$ is computed across lines within a single function, meaning the threshold is function-relative, but the paper does not discuss whether this normalization is appropriate across functions with vastly different perplexity distributions (a function consisting mostly of low-perplexity boilerplate vs. a function with complex, varying logic).

The consequence: Without validated block boundaries, the knapsack selection step (which selects subsets of blocks to retain) may be operating on units that do not correspond to meaningful code segments. If a block boundary is placed mid-statement (splitting a function call across two blocks), the knapsack could select one block without the other, producing syntactically invalid compressed code that the downstream LLM cannot interpret. If a block boundary fails to separate two semantically distinct regions (grouping a function signature with an unrelated implementation detail into the same block), the knapsack loses granularity and may be forced to retain or discard the combined unit as a whole, reducing the precision of fine-grained compression.

The ablation comparing perplexity-based chunking to line-based chunking (Table VII) shows only a 1.57 ES improvement (from 55.98 to 57.55), suggesting that the block boundaries are not providing a dramatically better segmentation signal than naive line splitting. This could mean either that (a) the perplexity boundaries are roughly as good as line boundaries for this task, (b) the knapsack selection is robust enough to compensate for poor segmentation, or (c) the downstream LLM is robust enough to handle slightly broken compressed code. The paper does not disentangle these possibilities. The 2.48 ES drop when replacing knapsack selection with random line selection (from 57.55 to 55.07) suggests that the selection mechanism matters more than the segmentation mechanism—but without rigorous evaluation of the segmentation quality itself, it is unclear whether a better segmentation method (e.g., AST-based with semantic merging) would unlock larger gains from the fine stage than the 1.45–2.48 ES improvements observed.

A more subtle consequence: the perplexity-based method computes unconditional perplexity for each line (no surrounding context), which means it cannot detect boundaries that depend on semantic context rather than local surprisal. A line like return result may have low unconditional perplexity (it's a common pattern), but in a function that normally returns None, this line represents a significant semantic shift. The unconditional perplexity signal would miss this boundary. Conversely, a line like x = some_obscure_function(y) may have high unconditional perplexity simply because some_obscure_function is rare, not because it represents a semantic boundary. The method conflates token rarity with semantic shift, which are correlated in many cases but not equivalent.

What evidence exists in the paper: The only validation of block boundary quality is the qualitative example in Figure 4, which shows perplexity spikes aligning with some structural transitions in the evaluate_blind function. This is a single example, chosen to illustrate the method, and cannot serve as systematic validation. The paper does not report: the average number of blocks per function across the evaluation datasets; the distribution of block sizes; any measure of boundary precision/recall against a ground-truth segmentation; any analysis of whether syntactically invalid compressed code is ever produced (e.g., blocks that are partial statements). The ablation comparing perplexity-based to line-based chunking (Table VII) provides indirect evidence that the former is better, but the 1.57 ES gap is measured on downstream task performance rather than segmentation quality, so it conflates segmentation quality with the downstream LLM's robustness to broken input.

The paper also does not specify $\alpha$, which makes the method unreproducible without guessing or re-implementing the sensitivity analysis from scratch. The natural language Meta-Chunking work (Zhao et al., 2024) likely specifies $\alpha$ for text, but the paper provides no guidance on whether the same value applies to code or how sensitivity to $\alpha$ varies.

Mitigation status: The paper does not acknowledge this as a limitation. Section VI-B (Discussion) focuses on a case study of successful compression and mentions failure modes only in terms of "context lacks information relevant to the task" or "ambiguous instruction"—not in terms of segmentation failures. The ablation in Table VII compares perplexity-based chunking only to line-based chunking, not to any syntax-aware alternative (AST-based, control-flow-based, or dependency-based segmentation). The authors do not suggest future work on validating or improving code segmentation methods. For a practitioner implementing LongCodeZip, the block boundary detection is effectively a black box—the paper provides the conceptual motivation but not the parameter values or validation results needed to trust or tune this component.

The Adaptive Budget Allocation Mechanism Uses a Fixed Importance Sensitivity Parameter with Unknown Robustness Across Context Sizes

The assumption or constraint: The adaptive budget allocation mechanism (Algorithm 1) distributes the fine-grained token budget across retained functions proportionally to their normalized AMI scores, using a sensitivity parameter $\beta$ that controls how much the allocation is biased toward high-AMI functions. The paper sets $\beta = 0.5$ for all three tasks (Section IV-E), determined "through experiments on a small held-out set that did not overlap with the test data." No sensitivity analysis of $\beta$ is reported—the paper does not show performance curves as $\beta$ varies from 0 (uniform allocation) to higher values (increasingly skewed toward top functions). The adaptive allocation is evaluated only against a "w/o Adaptive Budget Allocation" ablation (Table VII), which sets $\beta = 0$ effectively, showing a 2.34 ES drop. But whether $\beta = 0.5$ is optimal, or whether different tasks or context sizes would benefit from different $\beta$ values, is unknown.

The consequence: The fixed $\beta = 0.5$ setting encodes a specific assumption about how importance should translate into budget allocation: the most important function gets up to 50% more budget than the uniform baseline, and the least important gets up to 50% less (Equation 5: the multiplier is 1 + 0.5 * (2*AMInorm - 1), ranging from 0.5 to 1.5). This is a moderate redistribution—it biases allocation but does not starve low-AMI functions entirely. In scenarios where the context contains a small number of extremely important functions and a large number of marginally relevant ones (e.g., a single Config class among dozens of utility functions), a higher $\beta$ might be beneficial—directing nearly all budget to the critical functions and compressing the rest to near-zero. Conversely, in scenarios where relevance is relatively uniform across many selected functions (e.g., a module where many functions contribute to the overall logic), a lower $\beta$ might prevent over-compression of functions that are individually moderate-AMI but collectively important for coherence.

A more critical issue: $\beta$ interacts with the number of retained functions. If the coarse stage selects 5 functions (tight coarse budget), the normalized AMI scores span a limited range (the lowest retained function still had high enough AMI to make the cut), and $\beta = 0.5$ produces modest redistribution. If the coarse stage selects 50 functions (generous coarse budget), the lowest retained function may have AMI near zero (barely above the rejection threshold), and $\beta = 0.5$ would allocate it 50% less budget than the baseline—which on a very low baseline retention ratio (many functions competing for budget) could reduce it to near-zero tokens, effectively removing it. The adaptive allocation's behavior is therefore sensitive to the coarse budget, which varies with $R_{\text{fine}}$ and the total budget $B$. The paper's fixed $\beta$ across tasks with very different $R_{\text{fine}}$ values (0.8 for completion, 0.3 for summarization, 1.0 for RepoQA) means the effective redistribution magnitude varies across tasks even though $\beta$ is constant, but the paper does not analyze this interaction.

What evidence exists in the paper: The paper provides only a single ablation test: removing adaptive allocation entirely (Table VII, "w/o Adaptive Budget Allocation") and observing a 2.34 ES drop on code completion with Qwen2.5-Coder-7B. There is no sweep of $\beta$ values to identify the sensitivity of this effect, no analysis of whether $\beta = 0.5$ generalizes across the other tasks (summarization, RepoQA) or models, and no characterization of when the adaptive allocation matters most (e.g., at high compression ratios where budgets are tight, or when the AMI distribution across retained functions has high variance). The held-out set used for tuning $\beta$ is not described in terms of size, composition, or the range of $\beta$ values explored.

Mitigation status: The paper presents the consistent use of $\beta = 0.5$ across tasks as a feature (a single parameter that works across use cases) rather than as a limitation. The ablation demonstrates that removing adaptive allocation hurts, which validates the mechanism, but does not characterize the robustness of the chosen $\beta value. The authors do not suggest future work on making $\beta$ adaptive to context characteristics (e.g., computed from the variance of AMI scores) or learning it from data. For a practitioner, this means $\beta = 0.5$ is a reasonable default, but there is no guidance on when or how to tune it for new tasks, context sizes, or model families.

The Revision-Aware and Combined Search-Revision Extensions Are Not Explored, Leaving the Integration with Complementary Code Intelligence Techniques Undefined

The assumption or constraint: LongCodeZip is designed as a "plug-and-play" compression framework that produces standard text output consumable by any downstream LLM (Section III introduction). However, the paper evaluates LongCodeZip only in a single-pass, feed-once-and-generate paradigm: compress the context, concatenate with the instruction, feed to the LLM, and collect a single greedy/single-sample output. The paper explicitly notes in Section V-A that "RAG-based retrieval methods are complementary to our compression approach and could potentially be combined with our framework," and in Section VIII-B that LongCodeZip "provides broader applicability across diverse long-context code tasks" compared to retrieval-only methods. But neither integration is implemented or evaluated—no experiment combines LongCodeZip with retrieval, with iterative refinement, with multi-sample decoding, with self-consistency, or with any other test-time compute augmentation technique. The paper's cross-model results (Table VIII) show that the compression model and generation model can be different, but this transferability is used only to reduce compression cost, not to enable novel pipeline architectures.

The consequence: The paper's evaluation frames LongCodeZip as a standalone replacement for other context reduction methods (RAG, token-level compressors), but its practical deployment value may be highest as a component in a larger pipeline. For example: RAG retrieves candidate files from a repository → LongCodeZip compresses the retrieved context → the compressed context is fed to the LLM. Or: LongCodeZip compresses the context → the LLM generates a candidate → the generation is verified against the compressed context's constraints → if verification fails, the context is re-compressed with a different budget allocation and re-fed to the LLM. The paper provides no data on how LongCodeZip would perform in these composite architectures, leaving practitioners to guess about integration points.

A specific architectural question the paper does not address: should the AMI scores for compression be computed using the same model that will perform generation, or can they be computed once by a small, fast model and reused across multiple generation models? The cross-model results (Table VIII) answer this for the single-pass paradigm: yes, AMI from a small model works for a larger generation model. But in an iterative refinement setting—where the generation model produces an output, the verifier checks it, and the context is re-compressed for a second attempt—would the AMI scores need to be recomputed conditioned on the failed first attempt? The first attempt provides additional information about what the generation model found useful or confusing about the compressed context, which a more sophisticated system could use to refine the compression. LongCodeZip's current architecture has no mechanism for incorporating feedback from downstream generation into the compression decisions, but this is a natural extension that the paper neither implements nor discusses.

More pragmatically, the paper does not characterize how LongCodeZip interacts with different decoding strategies. All experiments use greedy or single-sample decoding. If the generation LLM is run with temperature > 0 and multiple samples are drawn (e.g., best-of-N or majority voting), would the compressed context produce less diverse outputs because fewer functions are available to inspire varied solutions? Would the verifier-based selection be biased by the compressed context in ways that a full context would not? These questions matter for production deployments where the generation model is run with non-zero temperature for creative or diverse outputs, but the paper provides no data to answer them.

What evidence exists in the paper: The paper contains no experiments combining LongCodeZip with any other technique. The cross-model experiments (Table VIII) are the closest the paper comes to integration studies, but they only vary the model used for compression while keeping the rest of the pipeline fixed. The discussion of RAG complementarity (Section V-A) is speculative rather than empirical. The paper's threat-to-validity discussion (Section VII) does not mention this as a limitation—it frames the evaluation as comprehensive but does not acknowledge the narrowness of the pipeline architecture tested.

Mitigation status: The paper acknowledges the complementarity with retrieval in qualitative discussion (Sections V-A, VIII-B) but treats it as a direction for future work rather than a gap in the current evaluation. The authors write: "RAG-based retrieval methods are complementary to our compression approach and could potentially be combined with our framework to further enhance performance by first retrieving relevant content and then applying our compression techniques" (Section V-A). This is a reasonable statement of future possibility, but it does not constitute evidence that the combination would outperform either method alone or that the integration is straightforward. The paper does not suggest specific combination architectures, potential failure modes of integration, or experiments to evaluate the combined approach.

7. Implications and Future Directions

How This Work Changes the Landscape

LongCodeZip shifts the conversation around context management for code LLMs from retrieval-centric to relevance-centric. The dominant paradigm for handling long code contexts has been retrieval-augmented generation: embed the instruction, embed candidate code snippets, measure cosine similarity, concatenate the top-k, and feed to the LLM. This pipeline answers the question "What code looks like the instruction?" LongCodeZip demonstrates—with a 7.89 ES point advantage over similarity-based ranking on code completion (Table VII)—that this is the wrong question. The right question is "What code helps the model predict the instruction?" Measured through approximated mutual information, this relevance signal captures functional dependencies (the Config class that supplies parameters to train_model) that embedding similarity systematically misses.

This is not a paradigm shift in the sense of a new architectural paradigm (transformers replacing RNNs) or a new training paradigm (pretraining plus fine-tuning). It is a reframing of the context selection problem from an external similarity judgment to an internal model introspection. The magnitude is significant but incremental: the paper does not introduce new model architectures, training objectives, or hardware requirements. It introduces a measurement methodology—how to ask a pretrained LM what context it finds useful—and demonstrates that this methodology, applied through a two-stage compression pipeline, produces practical efficiency gains (4.3–5.6× compression with preserved performance, 42% end-to-end latency reduction) that existing approaches cannot match.

The paper resolves or at least explains two contradictions in prior work. First, it explains why RAG works well on some code tasks and fails on others: tasks where relevant context shares surface vocabulary with the instruction (the get_email_by_id/get_account_by_id example in Figure 1) succeed; tasks where relevance is functional rather than lexical (the Config/train_model example) fail. The paper doesn't claim similarity is useless—it quantifies its ceiling (49.66 ES vs. 57.55 for AMI-based ranking, Table VII) and shows precisely where it breaks down. Second, it explains why prior code compressors underperform on long contexts: DietCode and SlimCode were designed for single-function compression, and their structural heuristics (attention-based token importance, rule-based pruning with program dependency graphs) don't scale to multi-file, repository-level contexts where cross-function dependencies dominate. The paper's 20–30 point accuracy advantage over these methods on RepoQA (Table IV) quantifies the gap.

Several research directions become more attractive as a result of this work. The demonstration that a 0.5B compression model produces AMI rankings nearly equivalent to a 7B model (Table VIII, average ES spread of <1.1 points across all compression-model/generation-model pairs) makes model-introspective context selection practically deployable—you don't need a large, expensive model to compute relevance scores, which removes the economic barrier that might have discouraged adoption. The finding that compression sometimes improves downstream performance (RepoQA: 75.3% compressed vs. 38.3% uncompressed on DeepSeek-Coder-6.7B, Table IV; summarization: 55.07 vs. 44.95 on Seed-Coder-8B, Table III) reframes compression from a cost-saving concession to a quality-improving preprocessing step, making it harder to justify not compressing in long-context code scenarios. And the ablation showing that the coarse-grained stage (function-level AMI ranking) accounts for ~80% of the method's advantage over similarity-based approaches (Table VII) tells future researchers where to focus: improving the relevance scoring function, not the fine-grained pruning mechanics.

Conversely, some research directions become less attractive. The paper's results on token-level compression for code are damning: LLMLingua achieves 1.5–8.7% accuracy on RepoQA (Table IV), and LLMLingua-2 achieves 21.56 ES on code completion with Qwen2.5-Coder-7B (Table II)—a 35-point collapse from the no-compression baseline. These numbers suggest that further refinements to token-level importance scoring for code are unlikely to yield practical gains; the fundamental problem is that code's non-local dependencies make token-level pruning structurally destructive regardless of how accurately token importance is estimated. Similarly, the paper's comparison with advanced RAG methods (A3-CodGen, cAST, RepoGenix, RLCoder in Table VI) shows that sophisticated retrieval strategies (structural chunking via AST, third-party library awareness, reinforcement-learned retrievers) are outperformed by a simple AMI-based function ranking at higher compression ratios. This doesn't mean retrieval research is dead—the paper explicitly frames retrieval as complementary—but it does suggest that improving the relevance signal (what the paper does) yields larger gains than improving the retrieval architecture (what the RAG methods do), and that future retrieval systems should incorporate model-introspective signals like AMI rather than relying solely on embedding similarity.

Follow-Up Research This Work Enables

Combining AMI-based compression with retrieval in a unified context selection pipeline. The paper states that RAG and LongCodeZip are complementary but never evaluates the combination. The natural experiment would be: on a repository-level code completion benchmark, compare (a) retrieval-only (RAG with UniXCoder), (b) compression-only (LongCodeZip on the full context), and (c) retrieval-then-compression (RAG retrieves top-K files/functions, LongCodeZip compresses the retrieved candidates). The hypothesis—untested in the current paper—is that retrieval-then-compression would outperform either alone: retrieval ensures the context contains the necessary information (addressing LongCodeZip's limitation when relevant code is absent), while compression removes noise from within the retrieved set (addressing RAG's limitation of including irrelevant snippets that rank high by similarity). A strong evaluation would measure performance across varying retrieval depth (K from 5 to 100) and compression budgets to identify the optimal pipeline configuration, and would include a cost accounting that sums retrieval embedding computation, AMI computation, and generation cost to determine the most cost-efficient pipeline for a given task.

Extending AMI-based relevance scoring to iterative and multi-turn code generation. The current paper evaluates LongCodeZip only in single-pass generation (compress once, generate once). Real-world code assistance is often multi-turn: a developer asks a question, the LLM generates an answer, the developer asks a follow-up, the LLM generates again, and the conversation accumulates context. Each turn provides new information about what the developer needs, which could refine the relevance estimates for the next compression pass. A natural extension would be: after the first generation, measure the LLM's perplexity on the generated code conditioned on different subsets of the original context (a "retrospective AMI"), and use this signal to re-compress the context for the next turn, dropping parts that proved irrelevant and expanding parts that the model struggled with. The experiment would measure whether multi-turn adaptive compression achieves higher cumulative task success than single-pass compression with a fixed budget, on a multi-turn code editing or debugging benchmark. This would also stress-test the paper's claim that AMI transfers across models—would retrospective AMI computed on the generation model itself outperform AMI computed by a smaller compression model, or does the transferability hold even for the specific generation just produced?

Training a lightweight AMI predictor to eliminate per-function forward passes. The dominant cost of LongCodeZip is computing perplexity for each function chunk against the instruction (Table IX: 2.58 seconds compression overhead). This is currently done with a full LM forward pass per function, scaling linearly with the number of functions. The cross-model results (Table VIII) show that a 0.5B model produces adequate AMI rankings, but a 0.5B model doing ~100 forward passes on a 10K-token context is still substantial. A cheaper approach: train a lightweight classifier (potentially a small transformer or even a bag-of-embeddings model with UniXCoder features) to predict AMI scores directly from function-instruction pairs, using the 7B model's AMI scores as training targets via distillation. The training data would be pairs of (function code, instruction) with continuous AMI labels, generated once offline for a diverse set of code tasks. The experiment would compare the predictor's AMI ranking correlation (Spearman's ρ against the gold 7B AMI ranking) and downstream task performance at various predictor sizes (from 10M to 500M parameters), measuring whether a distilled predictor can achieve the same downstream task performance as the full AMI computation at a fraction of the cost.

Characterizing the failure modes of perplexity-based block boundary detection and comparing against syntax-aware alternatives. The paper introduces perplexity-based block segmentation as a code-aware alternative to syntactic parsing but provides only a single qualitative example (Figure 4) and no quantitative validation. A stress-test would construct a ground-truth block segmentation dataset for code—perhaps using AST statement boundaries merged by control-flow coherence (e.g., group an if-header, its body, and its else-body into separate blocks; group consecutive assignments into blocks; separate function signatures from bodies)—and evaluate the perplexity-based method against this ground truth on precision, recall, and adjusted Rand index. The experiment would sweep the α sensitivity parameter (unreported in the paper) to identify the optimal value for code and measure whether optimal α varies by language, code style, or function length. It would also compare against a simple AST-based segmenter (split at statement boundaries, merge short adjacent statements) on both segmentation quality and downstream compression performance, to determine whether the LMs "understanding" of code structure (perplexity-based) offers advantages over formal syntactic structure (AST-based) for the specific purpose of context compression.

Stress-testing LongCodeZip on deliberately adversarial situations: missing context, ambiguous instructions, and distribution-shifted code. The paper acknowledges anecdotally (Section VI.B) that LongCodeZip struggles when context lacks relevant information or instructions are ambiguous, but provides no systematic evaluation. A diagnostic benchmark would construct three test suites: (1) Missing context: for each test query, randomly remove the most relevant function (as determined by a held-out oracle) from the provided context before compression, then measure whether LongCodeZip's downstream performance degrades gracefully (toward the No Context baseline) or catastrophically (to worse than No Context, because the model receives misleading partial context). (2) Ambiguous instructions: using the same code contexts, replace specific instructions ("Complete function X with signature Y") with increasingly vague versions ("Fix this code," "Improve this module") and measure the correlation between instruction specificity and AMI ranking quality (Spearman's ρ between AMI rankings from specific vs. vague instructions). (3) Distribution-shifted code: evaluate on code from domains unlikely to be well-represented in the compression model's pretraining data (e.g., esoteric programming languages, highly domain-specific DSLs, obfuscated code) to test whether AMI relevance scoring breaks when the LM doesn't understand the code's semantics. These experiments would define the boundary conditions for LongCodeZip's applicability more precisely than the current all-succeeding benchmark results.

Scaling analysis of compression benefit as generation model size increases. The paper evaluates generation models from 6.7B to 8B parameters (plus two commercial models of unknown size). An open question: does compression benefit diminish as models get larger? Larger models may handle long contexts better natively (the "lost in the middle" effect may be weaker), reducing the quality improvement from compression. Conversely, larger models are more expensive per token, making the cost savings from compression more economically valuable. A scaling experiment would fix compression hyperparameters and measure the performance gap between compressed and uncompressed contexts across a range of generation model sizes (e.g., DeepSeek-Coder variants at 1.3B, 6.7B, 33B; Qwen2.5-Coder at 0.5B, 1.5B, 3B, 7B, 14B, 32B) on the Long Code Completion and RepoQA benchmarks. The key output would be a curve showing ΔES (uncompressed minus compressed) as a function of model size—if the gap shrinks with model size, there is a crossover point beyond which compression provides cost savings but not quality improvement; if the gap remains constant or grows, compression is beneficial at all scales.

Practical Applications and Downstream Use Cases

IDE-integrated code completion with reduced latency and API costs. The most direct application is in developer tools that send context to cloud-based code LLMs for every keystroke or cursor movement. The paper shows that on the Long Code Completion task, compressing a 9,328-token context to ~2,000 tokens reduces generation latency from 15.7 seconds to 6.6 seconds (a 58% reduction, Table IX) while preserving or slightly improving completion quality (57.55 ES compressed vs. 56.36 ES uncompressed for Qwen2.5-Coder-7B, Table II). For a developer typing in an IDE with autocomplete firing every few seconds, a 9-second latency saving per completion directly translates to a more responsive experience. More critically for commercial API costs: a 4.3× input token reduction means 77% lower per-query API charges. At GPT-4o pricing (2.50–10permillioninputtokensdependingonbatch/context),asingle10K−tokenquerycosts2.50–10 per million input tokens depending on batch/context), a single 10K-token query costs 0.025–0.10; across thousands of daily completions for a development team, the savings compound to meaningful operational budget reductions. The cross-model transferability (Table VIII) means the compression can run on a small local model (0.5B parameters, fitting on a developer's laptop GPU) while the generation runs on a cloud API, keeping the compression latency low and avoiding additional API costs for the compression step itself.

Batch evaluation of code LLMs on long-context benchmarks. Researchers and companies evaluating code LLMs on benchmarks like RepoQA, LongCodeArena, or LongCodeBench currently pay the full inference cost for every test example. For a benchmark with 500 examples averaging 10K tokens of context, evaluating a single model costs ~5 million input tokens. Using LongCodeZip to compress contexts before evaluation—as the paper demonstrates on RepoQA where compressed DeepSeek-Coder-6.7B accuracy (75.3%) nearly doubles the uncompressed accuracy (38.3%, Table IV)—could simultaneously reduce evaluation cost and improve measured performance, making it a strictly dominant preprocessing step. The conservative approach would be to run both compressed and uncompressed evaluations to verify that compression preserves or improves on the specific benchmark, then adopt compression for all subsequent model evaluations on that benchmark. Given that the hyperparameters from the paper transfer across models (same configuration works for DeepSeek-Coder, Qwen2.5-Coder, Seed-Coder, GPT-4o, and Claude-3.7-Sonnet, Tables II, V), the compression configuration likely needs to be tuned per benchmark but not per model.

Repository-scale code understanding for code review and documentation generation. The paper's summarization results (Table III) demonstrate that compressing module-level context (10,810 tokens on average) to ~2,000–5,000 tokens preserves or improves summary quality: on Seed-Coder-8B, compressed CompScore is 55.07 vs. 44.95 uncompressed; on DeepSeek-Coder-6.7B, 28.01 vs. 19.09. This directly applies to automated code review tools that need to understand large pull requests (potentially thousands of lines across dozens of files) to generate meaningful review comments or documentation updates. A code review tool could compress each modified file's context using the instruction "summarize the changes and their impact on the module" to produce a focused summary for a human reviewer, or compress the entire PR context before feeding it to an LLM for automated review generation. The two-stage architecture is particularly well-suited here: the coarse stage identifies which functions in the codebase are affected by the changes, and the fine stage preserves the implementation details within those functions that are most relevant to understanding the change's implications.

Self-improvement data generation for code LLMs with reduced pipeline cost. When using LLMs to generate training data for themselves (e.g., generating solutions for coding problems, then fine-tuning on correct solutions), the input context for each generation instance is often large (repository-level context for realistic code completion). The paper's result that compression can improve downstream performance on challenging retrieval tasks (RepoQA with DeepSeek-Coder-6.7B: 75.3% vs. 38.3%, Table IV) suggests that applying LongCodeZip as a preprocessing step in a self-improvement pipeline could increase the yield of correct solutions while reducing the generation cost per instance. If a self-improvement pipeline generates 100K training instances with an average 10K-token context, compressing to 2K tokens saves ~800 million input tokens (roughly $2,000–8,000 in API costs at GPT-4o pricing) while potentially improving the quality of the generated solutions (since the model attends to a cleaner, higher-information-density context). The training-free nature of LongCodeZip means it can be inserted into any existing self-improvement pipeline without modifying the generation model or the training loop.

When to Prefer This Method

The paper explicitly positions LongCodeZip against several named alternatives and provides sufficient comparative data to define clear decision rules:

  • Prefer LongCodeZip over retrieval-based methods (RAG) when the codebase has deep functional dependencies that don't manifest as lexical overlap between the instruction and relevant context—for example, configuration classes used by training functions, type definitions referenced throughout a module, or inherited methods that the instruction doesn't explicitly name. The 7.89 ES gap between AMI-based and similarity-based ranking on code completion (Table VII) quantifies this advantage. The RepoQA results (Table IV) show the gap is largest when the task requires identifying specific functions by their behavior rather than their names.

  • Prefer LongCodeZip over token-level compression methods (LLMLingua, LLMLingua-2) when the downstream task requires syntactically valid code as input. Token-level compression produces catastrophic failures on code QA (1.5–8.7% accuracy on RepoQA, Table IV) and severe degradation on code completion (21.56 ES for LLMLingua vs. 56.36 uncompressed with Qwen2.5-Coder-7B, Table II). Any task where the LLM needs to parse, complete, or reason about code structure—as opposed to summarizing it in natural language—should avoid token-level compression.

  • Prefer LongCodeZip over no compression when the input context is long (>5,000 tokens) and either (a) generation latency matters, since 4.3× compression reduces generation time by 58% (Table IX), or (b) the generation model is known to suffer from "lost in the middle" effects on long contexts, since removing irrelevant functions can improve task performance beyond the uncompressed baseline (RepoQA: 75.3–87.2% compressed vs. 38.3–86.0% uncompressed, Table IV; summarization: 55.07 vs. 44.95 on Seed-Coder-8B, Table III).

  • Prefer retrieval-then-compression (not evaluated but implied by the paper's architectural discussion) when the provided context may not contain all necessary information—a common scenario in interactive developer tools where the user's query references code across the repository that wasn't all preloaded into the context window. The paper explicitly states that RAG methods are "complementary to our compression approach and could potentially be combined" (Section V-A). In this setting, retrieval handles the recall problem (ensuring the context contains relevant code), and LongCodeZip handles the precision problem (removing noise from within the retrieved set).

  • The paper provides no basis for preferring retrieval-only methods over LongCodeZip when the full context is available and fits within the LLM's context window. Retrieval's advantage is in reducing the initial context size before it reaches the LLM—but if the context is already available, LongCodeZip's AMI-based selection outperforms retrieval-based selection at identifying which parts are relevant (Tables II, IV, V, VI). The paper does not establish a regime where similarity-based selection is preferable to mutual-information-based selection when both have access to the same candidate context.