EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

🎯 Pitch

A simple SVD on compression error fails to recover task accuracy—until you project the error into activation eigenspace first. Without any fine-tuning, EoRA boosts a 3-bit LLaMA3-8B by over 10 points on ARC-Challenge, using just 64 calibration samples.

1. Executive Summary

This paper introduces EoRA (Eigenspace Low-Rank Approximation), a fine-tuning-free method that compensates for the accuracy loss of compressed LLMs by attaching lightweight, task-specific low-rank residual paths without modifying the compressed backbone weights. Evaluated on LLaMA2-7B/13B and LLaMA3-8B models compressed via SparseGPT pruning and GPTQ quantization, EoRA projects compression errors into the eigenspace of each layer's input activations—weighting error columns by their corresponding eigenvalues so that SVD prioritizes the most activation-critical error components—and achieves substantial accuracy recovery across language generation (WikiText2), commonsense reasoning (ARC-Challenge), and math reasoning (MathQA, GSM8K), including gains of 10.84% on ARC-Challenge, 6.74% on MathQA, and 11.45% on GSM8K for LLaMA3-8B quantized to 3-bit. The method completes in minutes using as few as 32–64 calibration samples per task, establishing that task-specific compression error can be compensated without gradient-based training only when the approximation error is aligned with the layer-wise compression loss through eigenspace projection rather than through naive SVD on the raw weight differences.

2. Context and Motivation

The Core Problem: Fixed Compression Formats Force a Rigid Accuracy–Efficiency Tradeoff

The fundamental problem this paper addresses is subtle but practically important: after you compress an LLM using post-training techniques like quantization or structured pruning, you are locked into a discrete, hardware-constrained compression format that leaves no room for fine-grained adjustment of the accuracy–efficiency tradeoff. The paper formulates this as a new problem they call customized compensation (Section 1, Section 2): given an already-compressed model, can you attach lightweight, task-specific low-rank residual paths that recover task accuracy without modifying the compressed weights, thereby enabling flexible, per-task control over the accuracy–latency tradeoff?

This matters because real-world deployment scenarios are heterogeneous. Different users have different accuracy requirements for different tasks, and a single compression decision—say, 2:4 structured sparsity because that is what NVIDIA GPUs accelerate, or exactly 4-bit integer quantization because that is the only format with an optimized kernel—forces every task to live with the same accuracy penalty. The paper gives a concrete illustrative example: "a user may wish to boost the accuracy of a 2:4 sparsity-pruned model on math reasoning tasks, accepting a modest increase in memory usage and inference latency in return" (Section 1). Under existing post-training compression paradigms, that user has no dial to turn—they either accept the compressed model's degraded math accuracy or deploy a less-compressed (and thus larger, slower) model for all tasks.

The critical constraint the paper imposes on itself is that the compressed weights must not be modified during compensation (Section 2). This is not an arbitrary restriction—it is what enables a single, general-purpose compressed backbone to be shared across all deployments, with only lightweight, task-specific low-rank modules loaded dynamically as needed. The paper points to existing multi-adapter inference frameworks like vLLM as infrastructure that already supports this pattern (Section 1, Figure 1). Without this constraint, every task would require its own separately fine-tuned compressed model, defeating the purpose of having a shared compressed backbone.

A Gap Between What Users Need and What Existing Methods Provide

To understand why this problem is genuine rather than contrived, it helps to walk through what a practitioner actually faces when deploying a compressed LLM. Post-training compression methods like GPTQ (Frantar et al., 2023) for quantization and SparseGPT (Frantar and Alistarh, 2023) for pruning have made it possible to compress billion-parameter models using only a small calibration dataset, without access to the original training data or budget for full retraining. These methods solve an engineering problem—they produce a smaller, faster model—but they leave an accuracy problem in their wake. The compressed model is worse than the original, sometimes dramatically so. For instance, Table 2 shows that LLaMA3-8B quantized to 3-bit drops from 50.42% to 20.90% on ARC-Challenge and from 36.23% to 0.45% on GSM8K—effectively destroying the model's math reasoning capability.

The practitioner now has a menu of bad options:

Accept the accuracy loss. This is what happens by default, and the paper's results show it can be catastrophic for aggressive compression.
Use a less aggressive compression format. This sacrifices the very efficiency gains that motivated compression in the first place.
Fine-tune the compressed model via LoRA or QLoRA (Dettmers et al., 2023) on each task of interest. This requires gradient-based training, hyperparameter tuning, and potentially hours of GPU time per task—not viable for rapid deployment to new tasks.
Use existing fine-tuning-free low-rank compensation methods like ZeroQuant-V2 (Yao et al., 2024), which applies SVD directly to the compression error $\Delta W = W - \hat{W}$ . The paper shows this is substantially suboptimal (Table 1, Table 2) because it ignores calibration data and thus cannot be task-specific.

The gap, then, is a fine-tuning-free method that is both task-specific (unlike naive SVD) and rapid (unlike LoRA fine-tuning). This is the precise niche EoRA targets.

Where Existing Approaches Fall Short

The paper identifies and systematically distinguishes its work from three classes of prior approaches, each with a specific weakness:

1. Naive SVD on Compression Error (ZeroQuant-V2)

The most direct baseline is to compute the compression error $\Delta W = W - \hat{W}$ for each linear layer, apply SVD to obtain a low-rank approximation, and attach the resulting matrices as a residual path. This is what ZeroQuant-V2 does, and it has the advantage of being fine-tuning-free and computationally cheap. However, the paper identifies a fundamental misalignment (Section 2, Equation 2 vs. Equation 1): naively minimizing the Frobenius norm of the error approximation,

$\arg\min_{B,A} \|\Delta W - BA\|_F$

does not minimize the actual layer-wise compression loss,

$\arg\min_{\hat{W}} \|W X - \hat{W} X\|_F$

because the input activations $X$ are completely absent from the optimization objective. The result is that the SVD approximation allocates its low-rank capacity uniformly across all error components, regardless of whether those components correspond to activation channels that are important for the task at hand. On tasks where those unimportant channels dominate the error norm, the SVD approximation wastes its budget on irrelevant errors while leaving task-critical errors uncompensated.

2. Activation-Statistics-Based Scaling Methods (Act-S, ASVD, LQER, SLiM, OATS, QERA)

A natural improvement over naive SVD is to scale the compression error by some function of the input activation magnitudes before applying SVD—the intuition being that error columns corresponding to high-magnitude activations (outlier channels) are more important and should be prioritized. The paper implements a representative version of this idea, which they call Act-S (Section 4.1), following the approach of ASVD (Yuan et al., 2023): construct a diagonal scaling matrix $S$ where each entry $S_{ii}$ is proportional to the square root of the average absolute activation value in channel $i$ , scale the error as $\Delta W'' = \Delta W S$ , apply SVD, then project back via $S^{-1}$ . The appendix (Section A.13) provides an extensive comparison with a family of similar methods—SLiM, OATS, LQER, QERA—all of which differ in how the scaling diagonal is constructed but share the same conceptual approach.

The paper's critique of these methods is twofold. First, and most directly, they are heuristic: there is no guarantee that scaling by activation magnitude minimizes the layer-wise compression loss in Equation 1. An error component in a high-magnitude activation channel is not necessarily more consequential for the final output than an error in a lower-magnitude but semantically critical channel. Second, as shown in Table 18, while these methods improve over naive SVD, they consistently underperform EoRA—the gap between EoRA and the best activation-statistics-based method on 3-bit LLaMA3-8B MathQA is 29.11% vs. 25.60% (LQER), for instance, a substantial margin.

3. Gradient-Based or Iterative Low-Rank Compensation (ApiQ, LRC, CALDERA, LoRA/QLoRA)

The paper also compares against methods that use more computation to achieve better compensation:

ApiQ (Liao et al., 2024) optimizes low-rank matrices using gradient-based training to minimize the layer-wise compression loss directly (Equation 6). This is conceptually the right objective, but as the paper notes, "the optimization time for both EoRA and Act-S is comparable, with each completing within minutes, whereas ApiQ requires over hours to optimize" (Section 4.1). The layer-wise variant of ApiQ is used for fair comparison since more resource-intensive variants "align more closely with PEFT methods rather than fine-tuning-free low-rank approximation approaches."
LoRA/QLoRA fine-tuning (Hu et al., 2021; Dettmers et al., 2023) keeps the compressed backbone frozen and trains low-rank adapters via gradient descent. The paper treats this as an orthogonal capability: EoRA can serve as a superior initialization for LoRA fine-tuning (Section 4.4, Table 3), but the fine-tuning-free setting is the primary contribution because it enables rapid task adaptation that gradient-based methods cannot match.
Iterative methods like LRC (Scetbon and Hensman, 2024) and CALDERA (Saha et al., 2024) update both the quantized weights and the low-rank matrices in an alternating fashion. The paper's key objection (Section A.13) is that these methods produce task-specific quantized models—they modify the compressed weights themselves, which breaks the shared-backbone paradigm that EoRA is designed to enable. "LRC requires iterative updates to weights and low-rank modules, leading to task-specific quantized models. In contrast, EoRA only adapts the low-rank modules, allowing a shared compressed backbone."

4. Training-Data-Dependent or Architecture-Specific Approaches

A broader class of methods—including quantization-aware training (QAT) variants like LR-QAT (Bondarenko et al., 2024) and SLoPe (Mozaffari et al., 2025), and fine-tuning methods like RILQ (Lee et al., 2025)—are distinguished by their reliance on gradient-based training or modifications to the pretraining pipeline. The paper explicitly positions these as belonging to a different regime: "In the LLM compression community, it is standard practice to distinguish between PTQ and QAT approaches, as they serve different purposes and are not typically benchmarked against each other" (Section A.13). EoRA is firmly a post-training method—it operates on an already-compressed model with no gradient computation—and the fair baselines are other post-training, fine-tuning-free methods.

The Theoretical Gap: Why Activation-Awareness Alone Isn't Enough

To appreciate what makes EoRA's approach non-obvious, it helps to understand why the natural next step—"just weight the SVD by activation magnitudes"—is insufficient. The paper provides a mathematical framework for this in Section 3, which we will explore in detail in the Technical Approach, but the motivation can be previewed here.

The layer-wise compression loss with low-rank compensation (Equation 5) is:

$\arg\min_{B,A} \|W X - (\hat{W} + BA) X\|_F = \arg\min_{B,A} \|\Delta W X - BA X\|_F$

which expands (Equation 6) to:

$\arg\min_{B,A} \left[\text{trace}\left((\Delta W - BA) X X^T (\Delta W - BA)^T\right)\right]^{1/2}$

The key observation is that $X X^T$ is the activation covariance matrix—an $k \times k$ matrix where $k$ is the input dimension of the layer. This matrix captures both the magnitudes and the correlations between activation channels. Naive activation-magnitude scaling (Act-S and its relatives) uses only the diagonal of this matrix (the per-channel variances), discarding all off-diagonal correlation information. But in transformer layers, activation channels are highly correlated—especially after attention and MLP operations—so the off-diagonal structure matters. EoRA's eigenspace projection via eigendecomposition $X X^T = Q \Lambda Q^T$ captures the full covariance structure, including these correlations, and weights the error approximation accordingly.

This is why the paper's results show EoRA outperforming Act-S not just by a small margin but by substantial amounts—for instance, 2.9%, 2.1%, and 10.7% on ARC-C, MathQA, and GSM8K respectively for 2:4 pruned LLaMA3-8B (Table 1). The activation covariance structure encodes task-specific information that simple per-channel magnitude scaling cannot capture.

How the Paper Positions Itself

The paper positions EoRA at the intersection of three research threads—post-training compression, low-rank decomposition, and parameter-efficient fine-tuning—while carving out a precise, previously unoccupied niche (Section 5, Related Works):

From post-training compression: EoRA adopts the paradigm of using minimal calibration data and no gradient updates, making it compatible with any compression method (SparseGPT, GPTQ, Wanda—demonstrated in Sections 4.2.1 and A.4). It is not itself a compression method but a compensation method that augments an already-compressed model.
From low-rank decomposition: EoRA uses SVD as its computational engine but applies it to the projected compression error rather than to the weight matrix itself (as in ASVD, SVD-LLM) or to the raw error (as in ZeroQuant-V2). The theoretical contribution is proving that this projection makes SVD minimize the layer-wise compression loss rather than the error approximation loss—a guarantee that no prior fine-tuning-free method provides (Theorem 1, Section 3).
From parameter-efficient fine-tuning: EoRA adopts the residual low-rank adapter architecture familiar from LoRA but, critically, determines the adapter weights analytically (via eigendecomposition + SVD) rather than through gradient-based training. This makes it a fine-tuning-free method in the PEFT architectural tradition, and the paper shows it can also serve as a superior initialization when fine-tuning is eventually applied (Section 4.4).

The paper explicitly distinguishes itself from SVD-LLM (Wang et al., 2025), which is "conceptually close" (Section 5) because it also tries to align the SVD compression error with the layer-wise compression loss. However, SVD-LLM "relies on the matrix product of the activation being positive-definite, a condition often unmet in practice"—enforcing this requires modifications that introduce noise—whereas EoRA's eigendecomposition "only requires the matrix product to be symmetric—a property that naturally holds." This is a subtle but meaningful distinction: eigendecomposition is always well-defined for symmetric matrices, whereas positive-definiteness (required for Cholesky decomposition-based approaches like SVD-LLM) may fail if the activation covariance is numerically rank-deficient due to small eigenvalues, which commonly occurs in practice.

In summary, the paper identifies a specific, practically motivated gap: compressed LLMs have task-specific accuracy loss, existing fine-tuning-free compensation methods (naive SVD, activation-statistics scaling) are suboptimal because they don't align with the true compression loss, and gradient-based methods (ApiQ, LoRA, iterative compensation) are too slow or modify the backbone. EoRA fills this gap by providing the first fine-tuning-free method that is provably aligned with the layer-wise compression loss through eigenspace projection, completing in minutes with minimal calibration data.

3. Technical Approach

This is primarily a theoretical-methods paper that contributes a closed-form, fine-tuning-free algorithm for approximating compression errors in LLMs using low-rank matrices, where the core idea is that projecting the compression error into the eigenspace of the layer's input activations before applying SVD guarantees that the low-rank approximation minimizes the actual layer-wise compression loss rather than just the raw error norm.

3.1 Reader Orientation

EoRA is a compensation algorithm — not a compression algorithm itself — that takes an already-compressed LLM and attaches small, task-specific low-rank residual matrices to each linear layer such that the combined model (compressed weights + low-rank residuals) better approximates the original uncompressed model's behavior on a specific downstream task. The problem it solves is that existing fine-tuning-free approaches optimize the wrong thing: they approximate the weight difference $\Delta W = W - \hat{W}$ directly, which ignores which input channels actually matter for the task. EoRA's solution shape: use a few dozen calibration samples to compute the activation covariance, decompose it into eigenvectors and eigenvalues, project $\Delta W$ into that eigenspace (which implicitly weights error components by their task-specific importance), apply SVD, then project back — all in closed form, taking minutes, with no backpropagation.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that operate sequentially, one linear layer at a time:

Compressed Model Loader — provides the compressed weight matrix $\hat{W}$ and the original uncompressed weight matrix $W$ for each linear layer. This is fixed and shared across all tasks. Output: the raw compression error $\Delta W = W - \hat{W}$ .
Calibration Data Pass & Activation Collector — runs the compressed model on a small set of task-specific calibration sentences (e.g., 32–128 concatenated sequences of length 2048), collecting the input activations $X$ at each layer. These activations encode what input patterns the task actually presents to each layer. Output: for each layer $l$ , a matrix $\tilde{X} \in \mathbb{R}^{k \times n}$ representing the average input activations over the $n$ calibration tokens, where $k$ is the layer's input dimension.
Eigenspace Projection & SVD Engine (the core EoRA algorithm) — for each layer: (a) forms the activation covariance $\tilde{X}\tilde{X}^T$ , (b) runs eigendecomposition to obtain eigenvectors $Q$ and eigenvalues $\Lambda$ , (c) constructs the projection matrix $Q' = Q\sqrt{\Lambda}$ , (d) projects $\Delta W$ into the eigenspace as $\Delta W' = \Delta W Q'$ , (e) applies $r$ -rank SVD to $\Delta W'$ to obtain $B' = U'\Sigma'$ and $A' = V'^T$ , (f) projects $A'$ back to the original space as $A = A' Q'^{-1}$ . This produces two low-rank matrices per layer: $B' \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ .
Inference Engine with Optional Kernel Fusion — at inference time, each layer computes $\hat{W}X + B'AX$ , where $B'A = B'A' Q'^{-1}$ approximates the task-specific compression error. A custom CUDA kernel fuses the $B'$ multiplication with the quantized $\hat{W}$ kernel to reduce memory traffic (described in Section 4.5, Appendix A.6).

Information flows: calibration data → compressed model forward pass → per-layer activation collection → per-layer eigendecomposition + error projection + SVD → per-layer low-rank residual matrices → stored as task-specific adapters → dynamically loaded at inference alongside the shared compressed backbone.

3.3 Roadmap for the Deep Dive

First, the layer-wise compression loss with compensation (Equations 1 and 5–6), because this defines what "good compensation" mathematically means and sets up the optimization problem EoRA solves.
Second, the naive SVD baseline and why it fails (Equation 2), because understanding the failure mode motivates every design choice in EoRA.
Third, the eigenspace projection mechanism (the eigendecomposition of $\tilde{X}\tilde{X}^T$ , the construction of $Q'$ , the projection of $\Delta W$ ), because this is the central technical contribution and must be understood before Theorem 1.
Fourth, Theorem 1 and its proof, because this is the theoretical guarantee that distinguishes EoRA from all heuristic activation-scaling methods — it shows why eigenspace projection makes SVD minimize the right thing.
Fifth, the complete EoRA algorithm (Algorithm 1) and the reconstruction of $A$ from $A'$ , including the invertibility argument for $Q'$ , because this is the operational recipe.
Sixth, practical considerations: calibration data construction, rank selection, inference-time computation, and the relationship to LoRA fine-tuning, because these are what make the method deployable.

3.4 Detailed, Sentence-Based Technical Breakdown

We build up EoRA from first principles, starting with what post-training compression optimization actually minimizes, then showing why naive compensation fails, and finally constructing the eigenspace solution one logical step at a time.

The Layer-Wise Compression Loss (What We Actually Want to Minimize)

Post-training compression methods like GPTQ and SparseGPT are formulated as layer-wise optimization problems. For a single linear layer with original weight $W \in \mathbb{R}^{d \times k}$ (where $d$ is the output dimension and $k$ is the input dimension), the compression process finds a compressed weight $\hat{W} \in \mathbb{R}^{d \times k}$ that minimizes:

$\arg\min_{\hat{W}} \|W X - \hat{W} X\|_F$

where $X \in \mathbb{R}^{k \times n}$ is the input activation matrix for that layer (computed from the calibration data, with $n$ being the number of calibration tokens), and $\|\cdot\|_F$ denotes the Frobenius norm (the square root of the sum of squared entries — essentially the Euclidean norm generalized to matrices).

What it computes: For each of the $n$ calibration tokens, the layer produces a $d$ -dimensional output vector under both the original weight $W$ and the compressed weight $\hat{W}$ . The Frobenius norm aggregates the squared differences across all $n$ tokens and all $d$ output dimensions into a single scalar. Minimizing this scalar means finding the compressed weights that make the layer's outputs as similar as possible to the original layer's outputs on the calibration data.

Why this form: This formulation decomposes the global compression problem into independent per-layer subproblems, which is what makes post-training compression tractable for billion-parameter models — you don't need to backpropagate through the entire network. Each layer is optimized greedily to match its original output distribution, under the assumption (borne out empirically) that if each layer individually matches, the full model's behavior will be largely preserved.

When the compressed model is later evaluated on a task-specific dataset, the input activations $X$ are different from the calibration activations used during compression. This is where accuracy degradation comes from: the compressed weights were optimized for a generic calibration distribution, not the task-specific one.

Now, the customized compensation problem: given $\hat{W}$ (fixed — we cannot modify it), find low-rank matrices $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ (with $r \ll \min(d, k)$ ) that, when added as a residual path, minimize the task-specific layer-wise compression loss:

$\arg\min_{B, A} \|W X - (\hat{W} + BA) X\|_F = \arg\min_{B, A} \|\Delta W X - BA X\|_F$

where $\Delta W = W - \hat{W} \in \mathbb{R}^{d \times k}$ is the compression error — the element-wise difference between the original and compressed weights — and $X$ now comes from the task-specific calibration data. The equality holds because $WX - (\hat{W} + BA)X = (W - \hat{W})X - BAX = \Delta W X - BAX$ .

What this equation means operationally: We want to find matrices $B$ and $A$ whose product $BA$ approximates the compression error $\Delta W$ , but critically, the approximation is judged not by how close $BA$ is to $\Delta W$ in weight space (which would be $\|\Delta W - BA\|_F$ ), but by how close the activations produced by the compensated model are to the original model's activations. The $X$ matrix acts as a weighting: error in weight rows/columns that correspond to input dimensions that are consistently large or important for the task gets penalized more heavily than error in weight rows/columns that correspond to input dimensions that are rarely activated.

Using the identity that the squared Frobenius norm of a matrix equals the trace of its Gram matrix (Sun, 1991; cited as reference [14] in the paper), this objective can be rewritten in a form that exposes the role of the activation covariance:

$\arg\min_{B, A} \|\Delta W X - BA X\|_F = \arg\min_{B, A} \left[\text{trace}\left((\Delta W - BA) X X^T (\Delta W - BA)^T\right)\right]^{1/2}$

where $X X^T \in \mathbb{R}^{k \times k}$ is the (unnormalized) covariance matrix of the input activations, and the trace sums the diagonal elements of the matrix inside the brackets. The square root (exponent 1/2) converts the squared Frobenius norm (which the trace computes) back to the Frobenius norm itself.

What this form reveals: The term $(\Delta W - BA) X X^T (\Delta W - BA)^T$ is a quadratic form. For any candidate compensation $BA$ , its error is evaluated by: (1) computing the residual $\Delta W - BA$ , (2) multiplying by the activation covariance $X X^T$ on the right (which weights columns of the residual by their importance in activation space), (3) multiplying by the residual transpose on the left, and (4) taking the trace (sum of diagonal entries, which represents the total weighted error). The matrix $X X^T$ encodes both the per-channel activation magnitudes (on its diagonal) and the correlations between channels (on its off-diagonal). This is the object that naive SVD ignores.

This equivalence — from Equation 5 to Equation 6 — is the foundation on which the entire EoRA method is built, so it is worth pausing to verify: the Frobenius norm squared of a matrix $M$ is $\|M\|_F^2 = \text{trace}(M M^T)$ . Setting $M = \Delta W X - BA X = (\Delta W - BA) X$ , we have $M M^T = (\Delta W - BA) X X^T (\Delta W - BA)^T$ . The trace of this matrix is the squared Frobenius norm, and the square root recovers the norm itself. This is a standard matrix identity.

Why Naive SVD Fails (Equation 2 vs. Equation 1)

The simplest fine-tuning-free approach to obtain low-rank compensation is to directly apply SVD to the compression error $\Delta W$ . This is what ZeroQuant-V2 does. The optimization is:

$\arg\min_{B, A} \|\Delta W - B A\|_F$

This is a pure matrix approximation problem: find rank- $r$ matrices $B$ and $A$ whose product is as close as possible to $\Delta W$ in Frobenius norm. By the Eckart-Young theorem (Eckart and Young, 1936; cited as reference [16]), the optimal solution is given exactly by the truncated SVD of $\Delta W$ : if $\Delta W = U \Sigma V^T$ is the full SVD, then the rank- $r$ approximation is $U_r \Sigma_r V_r^T$ , where $U_r$ and $V_r$ contain the first $r$ left and right singular vectors (corresponding to the $r$ largest singular values in $\Sigma_r$ ), and we can set $B = U_r \Sigma_r$ , $A = V_r^T$ .

What it computes: The SVD finds the directions in weight space along which the compression error $\Delta W$ varies most, and allocates the limited rank budget to represent those directions first. It is the optimal low-rank approximation of $\Delta W$ in the Frobenius norm sense, but it is completely blind to which of those error directions correspond to important input channels for a specific task.

Why this fails for task-specific compensation: The issue is the missing $X X^T$ term. Compare the naive objective $\|\Delta W - BA\|_F$ with the true objective $\left[\text{trace}((\Delta W - BA) X X^T (\Delta W - BA)^T)\right]^{1/2}$ . In the true objective, error in row $i$ of $\Delta W - BA$ is weighted by $(X X^T)_{ii}$ (the variance of input channel $i$ ) plus cross-terms from correlated channels. In the naive objective, all rows are weighted equally. If the activation covariance $X X^T$ is far from the identity — which it is in practice, especially after layer normalization and attention operations that induce strong channel correlations — then the SVD-optimal approximation in weight space can be substantially suboptimal in activation space.

As the paper states in Section 2: "naively applying SVD to optimize error approximation loss (Eq.2) does not ensure minimization of the layer-wise compression loss (Eq.1) and ignores calibration data, making it ineffective for task-specific accuracy recovery."

This is not a hypothetical problem. The empirical evidence in Tables 1 and 2 shows ZeroQuant-V2 consistently underperforming EoRA by substantial margins, especially on aggressively compressed models where the compression error is large and the misalignment between weight-space and activation-space importance is most consequential.

The Eigenspace Projection: What, How, and Why

The central insight of EoRA is that the true objective (Equation 6) contains $X X^T$ as a weighting matrix, and that this matrix can be decomposed to reparameterize the problem so that applying SVD after the reparameterization becomes equivalent to minimizing the true objective. The mechanism is:

Step 1: Form the activation covariance. Given the task-specific calibration data, run the compressed model forward and collect the input activations to each layer. Compute $\tilde{X} \in \mathbb{R}^{k \times n}$ , the matrix of (average) input activations over the calibration set. The paper states: " $\tilde{X} \in \mathbb{R}^{k \times n}$ is the average of the input activations over the task-specific calibration set" (Section 3). Form the symmetric positive semi-definite matrix $\tilde{X}\tilde{X}^T \in \mathbb{R}^{k \times k}$ . This is the (unnormalized) activation covariance matrix — each entry $(i,j)$ is the dot product of the activation vectors at input channels $i$ and $j$ across the $n$ calibration tokens.

Step 2: Eigendecomposition. Perform eigendecomposition on $\tilde{X}\tilde{X}^T$ :

$\tilde{X}\tilde{X}^T = Q \Lambda Q^T$

where $Q \in \mathbb{R}^{k \times k}$ is an orthogonal matrix whose columns are the eigenvectors of $\tilde{X}\tilde{X}^T$ , and $\Lambda \in \mathbb{R}^{k \times k}$ is a diagonal matrix whose diagonal entries are the corresponding eigenvalues, sorted in descending order ( $\Lambda_{11} \geq \Lambda_{22} \geq \dots \geq \Lambda_{kk} \geq 0$ ).

What it computes: Eigendecomposition diagonalizes the activation covariance. The eigenvectors $Q$ form a new orthonormal basis for the input space — the "eigenspace" of the activations — where the basis vectors are ordered by how much variance the input activations exhibit along each direction. The first eigenvector (column 1 of $Q$ ) points in the direction of maximum activation variance across the calibration data; the second points in the orthogonal direction with the second-highest variance; and so on. The eigenvalue $\Lambda_{ii}$ measures the total squared activation magnitude (variance) along the $i$ -th eigenvector direction.

Why this form: Unlike the raw activation matrix, which mixes information across channels, the eigendecomposition cleanly separates the input space into independent components ordered by importance. This is analogous to Principal Component Analysis (PCA): the first few eigenvectors capture the dominant patterns in the activation data, and the corresponding eigenvalues quantify their importance. The paper explicitly draws this analogy: "Inspired by the classical Principal Component Analysis (PCA) algorithm, we leverage the eigenvalues of each activation channel as importance scores" (Section 3). The key property is that $\tilde{X}\tilde{X}^T$ is always symmetric (by construction), so its eigendecomposition always exists with real eigenvalues and an orthogonal $Q$ — unlike methods requiring positive-definiteness (e.g., SVD-LLM, which uses Cholesky decomposition), which can fail when eigenvalues are numerically zero.

Step 3: Construct the projection matrix. Reformulate the eigendecomposition as:

$Q \Lambda Q^T = (Q \sqrt{\Lambda})(\sqrt{\Lambda} Q^T) = Q' Q'^T$

where $Q' = Q \sqrt{\Lambda}$ and $\sqrt{\Lambda}$ is the diagonal matrix with $\sqrt{\Lambda_{ii}}$ on the diagonal. This factorization is possible because $\Lambda = \sqrt{\Lambda} \sqrt{\Lambda}$ and $\sqrt{\Lambda} = \sqrt{\Lambda}^T$ (diagonal matrices are symmetric).

What this means: The matrix $Q' \in \mathbb{R}^{k \times k}$ is a linear transformation constructed from the eigenstructure of the activation covariance. Its columns are the eigenvectors scaled by the square roots of the corresponding eigenvalues: column $i$ of $Q'$ is $\sqrt{\Lambda_{ii}}$ times column $i$ of $Q$ . This means that directions with high activation variance (large $\Lambda_{ii}$ ) are amplified in $Q'$ , while directions with low activation variance (small $\Lambda_{ii}$ ) are attenuated.

Why this particular construction: The scaling by $\sqrt{\Lambda}$ is the crucial step that makes the subsequent SVD minimize the true objective. If one were to use only $Q$ (without the $\sqrt{\Lambda}$ scaling), the projection would simply rotate the error into the eigenbasis without reweighting. The $\sqrt{\Lambda}$ factor embeds the eigenvalue importance weighting directly into the projected error matrix, so that when SVD later minimizes the Frobenius norm of the projected error, it automatically allocates more approximation capacity to the directions that matter more for the task.

Step 4: Project the compression error. Project $\Delta W$ into the eigenspace using $Q'$ :

$\Delta W' = \Delta W Q' \in \mathbb{R}^{d \times k}$

This is a right-multiplication, so the $k$ columns of $\Delta W$ (each corresponding to one input dimension) are linearly combined according to $Q'$ . Specifically, column $j$ of $\Delta W'$ is a weighted combination of columns of $\Delta W$ , with the weights given by row $j$ of $Q'$ .

What it computes: The projected error $\Delta W'$ represents the compression error in the eigenspace basis, scaled by activation importance. A column of $\Delta W$ corresponding to a high-activation-variance direction in the original input space gets amplified in $\Delta W'$ ; a column corresponding to a near-zero-variance direction gets attenuated. The result is that the Frobenius norm of $\Delta W'$ is dominated by the error components that matter most for the task.

Step 5: Apply SVD to the projected error. Run rank- $r$ truncated SVD on $\Delta W'$ :

$\text{SVD}(\Delta W') \approx U' \Sigma' V'^T$

where $U' \in \mathbb{R}^{d \times r}$ contains the top- $r$ left singular vectors, $\Sigma' \in \mathbb{R}^{r \times r}$ is diagonal with the top- $r$ singular values (in descending order), and $V' \in \mathbb{R}^{k \times r}$ contains the top- $r$ right singular vectors. Set $B' = U' \Sigma' \in \mathbb{R}^{d \times r}$ and $A' = V'^T \in \mathbb{R}^{r \times k}$ .

At this point, $B' A' \approx \Delta W'$ is the optimal rank- $r$ approximation of the projected error in the standard Frobenius norm (by the Eckart-Young theorem). But we need the approximation in the original space, not the eigenspace.

Step 6: Project back to original space. To recover an approximation of $\Delta W$ from the approximation of $\Delta W'$ , multiply by $Q'^{-1}$ on the right:

$\Delta W \approx B' A' Q'^{-1}$

since $\Delta W = \Delta W' Q'^{-1}$ (because $\Delta W' = \Delta W Q'$ and $Q'$ is invertible). The invertibility argument is:

$Q'^{-1} = \sqrt{\Lambda}^{-1} Q^T$

and $Q' Q'^{-1} = Q \sqrt{\Lambda} \sqrt{\Lambda}^{-1} Q^T = Q Q^T = I$ (since $\sqrt{\Lambda} \sqrt{\Lambda}^{-1} = I$ and $Q$ is orthogonal, so $Q Q^T = I$ ). All eigenvalues of $\tilde{X}\tilde{X}^T$ are non-negative, and assuming none are exactly zero, $\sqrt{\Lambda}^{-1}$ is well-defined. (In practice, near-zero eigenvalues may cause numerical instability; the paper does not discuss regularization for this, but the results suggest it is not a problem for the models and tasks studied.)

Now, the key simplification: the product $A' Q'^{-1}$ can be precomputed into a single matrix $A \in \mathbb{R}^{r \times k}$ , since both are known after the SVD step and $Q'^{-1}$ does not depend on the SVD result:

$A = A' Q'^{-1} = V'^T \sqrt{\Lambda}^{-1} Q^T$

This means the final compensation uses matrices $B'$ (from the eigenspace SVD) and $A = A' Q'^{-1}$ (the projected-back right singular vectors), and the forward pass becomes $\hat{W}X + B' A X$ . The dimensions work out: $B' \in \mathbb{R}^{d \times r}$ , $A \in \mathbb{R}^{r \times k}$ , so $B'A \in \mathbb{R}^{d \times k}$ — the same shape as $W$ and $\hat{W}$ , as required for a residual addition.

Why this projection-back step preserves efficiency: The product $A' Q'^{-1}$ is a $(r \times k)$ times $(k \times k)$ multiplication, but since $r \ll k$ , it can be done efficiently. More importantly, since $Q$ is orthogonal and $\Lambda$ is diagonal, $Q'^{-1} = \sqrt{\Lambda}^{-1} Q^T$ involves only: (1) a transpose of $Q$ , (2) an element-wise scaling by $1 / \sqrt{\Lambda_{ii}}$ , and (3) a matrix multiplication by $A'$ . The result $A$ has exactly the same dimensions as $A'$ ( $r \times k$ ), so there is no additional inference-time overhead compared to the naive SVD approach — the low-rank matrices consumed at inference are still $B'$ (size $d \times r$ ) and $A$ (size $r \times k$ ).

Theorem 1: Why Eigenspace Projection + SVD Minimizes the True Objective

The theoretical guarantee that justifies EoRA's design is Theorem 1 (Section 3), which states:

"For an activation matrix $X$ , whose matrix product $X X^T$ has an eigendecomposition given by $X X^T = Q \Lambda Q^T$ . By projecting the compression error $\Delta W$ into the eigenspace with $Q \sqrt{\Lambda}$ as $\Delta W' = \Delta W Q \sqrt{\Lambda}$ , minimizing Eq. 3 via SVD becomes equivalent to minimizing Eq. 6."

Eq. 3 is the EoRA loss $\|\Delta W' - B'A' \|_F$ , and Eq. 6 is the true task-specific layer-wise compression loss $\left[\text{trace}((\Delta W - BA) X X^T (\Delta W - BA)^T)\right]^{1/2}$ .

The proof works by direct algebraic manipulation, transforming the true objective into a form that is exactly the Frobenius norm of something that depends on $\Delta W'$ .

Starting from the true objective (Eq. 6), substitute $\tilde{X}\tilde{X}^T = Q \Lambda Q^T$ :

$\left[\text{trace}\left((\Delta W - BA) Q \Lambda Q^T (\Delta W - BA)^T\right)\right]^{1/2}$

Step 1: Absorb the $Q$ factor into the residual. Since $\Lambda Q^T (\Delta W - BA)^T = (\Delta W Q \Lambda - BA Q \Lambda)^T$ , and using the cyclic property of the trace ( $\text{trace}(MNP) = \text{trace}(PMN) = \text{trace}(NPM)$ for compatible matrices), the expression inside the trace can be rewritten. More carefully, since $Q \Lambda Q^T = (Q \sqrt{\Lambda})(\sqrt{\Lambda} Q^T)$ , we have:

$\left[\text{trace}\left((\Delta W - BA) (Q \sqrt{\Lambda})(\sqrt{\Lambda} Q^T) (\Delta W - BA)^T\right)\right]^{1/2}$

$= \left[\text{trace}\left((\Delta W Q \sqrt{\Lambda} - BA Q \sqrt{\Lambda})(\Delta W Q \sqrt{\Lambda} - BA Q \sqrt{\Lambda})^T\right)\right]^{1/2}$

This step uses: $(M)(N)(N^T)(M^T) = (MN)(MN)^T$ for $M = \Delta W - BA$ and $N = Q\sqrt{\Lambda}$ . Distributing: $M N = \Delta W Q \sqrt{\Lambda} - BA Q \sqrt{\Lambda}$ .

Step 2: Recognize $\Delta W'$ . Let $Q' = Q \sqrt{\Lambda}$ , so $\Delta W' = \Delta W Q'$ by definition. The expression becomes:

$= \left[\text{trace}\left((\Delta W' - BA Q')(\Delta W' - BA Q')^T\right)\right]^{1/2}$

Step 3: Convert trace-of-Gram back to Frobenius norm. Since $\text{trace}(M M^T) = \|M\|_F^2$ , and the square root of the trace gives the norm:

$= \|\Delta W' - BA Q'\|_F$

Step 4: Reparameterize. Define $B' = B$ and $A' = A Q'$ . Then $B' A' = B A Q'$ , and the objective becomes:

$\|\Delta W' - B' A'\|_F$

which is exactly the EoRA loss (Eq. 3). The minimization of this Frobenius norm over rank- $r$ matrices $B'$ and $A'$ is achieved, by the Eckart-Young theorem, by the truncated SVD of $\Delta W'$ : set $B' = U' \Sigma'$ and $A' = V'^T$ from $\text{SVD}(\Delta W')$ . The mapping back to the original parameterization is $B = B'$ and $A = A' Q'^{-1}$ , giving $BA = B' A' Q'^{-1}$ , which is the rank- $r$ approximation of $\Delta W$ that minimizes the true objective.

What this proof establishes: The equivalence is exact, not approximate. Minimizing the naive Frobenius norm of $\Delta W' - B'A'$ in the eigenspace is mathematically identical to minimizing the activation-weighted compression loss in the original space. The projection by $Q' = Q\sqrt{\Lambda}$ serves as a change of variables that absorbs the $X X^T$ weighting into the error matrix itself, so that an unweighted SVD on the transformed error becomes a weighted approximation in the original space.

Why this matters compared to heuristic scaling: Methods like Act-S compute a diagonal scaling $S$ based on per-channel activation magnitudes and then apply SVD to $\Delta W S$ , effectively assuming $X X^T$ is diagonal. This ignores off-diagonal correlations between activation channels. EoRA's eigendecomposition captures the full eigenstructure, including these correlations. When activation channels are correlated (which they are in practice, especially after self-attention layers that mix information across positions), the eigenvectors of $X X^T$ are not aligned with the original input basis, and a diagonal scaling is insufficient. Theorem 1 guarantees that EoRA's full eigenprojection handles this correctly.

The Complete EoRA Algorithm (Algorithm 1)

Algorithm 1 in the paper provides the step-by-step procedure. Let us walk through it with explicit dimensions and computational considerations.

Inputs:

$\tilde{X} \in \mathbb{R}^{k \times n}$ : The average input activations for this layer over the calibration set. The paper specifies that $\tilde{X}$ is the average (mean) of the activations, not the concatenation of all activations. This is important because it reduces the size of the eigendecomposition from depending on $n$ to depending only on $k$ , and it makes $X X^T$ a $k \times k$ matrix regardless of calibration set size.
$W \in \mathbb{R}^{d \times k}$ : The full-precision original weight.
$\hat{W} \in \mathbb{R}^{d \times k}$ : The compressed weight.
$r$ : The desired compensation rank (e.g., 64, 128, 256, 512).

Outputs: $B' \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ .

Algorithm steps:

1. Compute the compression error: $\Delta W = W - \hat{W} \in \mathbb{R}^{d \times k}$ This is a straightforward element-wise subtraction. The result is a dense matrix representing the weight values lost during compression.

2. Eigendecomposition of the activation covariance: $\tilde{X}\tilde{X}^T = Q \Lambda Q^T$ where $\tilde{X}\tilde{X}^T \in \mathbb{R}^{k \times k}$ is symmetric positive semi-definite. The eigendecomposition is computed using standard numerical linear algebra (e.g., torch.linalg.eigh for symmetric matrices, which is more numerically stable than general eig). The output is an orthogonal matrix $Q \in \mathbb{R}^{k \times k}$ and a diagonal matrix $\Lambda \in \mathbb{R}^{k \times k}$ .

For a typical transformer linear layer, $k$ ranges from 4096 (for LLaMA2-7B's attention projections) to 14336 (for LLaMA3-8B's intermediate FFN up-projection). The eigendecomposition of a $k \times k$ symmetric matrix has complexity $O(k^3)$ , which for $k = 14336$ is approximately $3 \times 10^{12}$ floating-point operations — substantial but manageable on a modern GPU, and done only once per layer per task. This is why the method takes "a few minutes" rather than seconds.

3. Form the projection matrix and its inverse: $Q' = Q \sqrt{\Lambda} \in \mathbb{R}^{k \times k}$ $Q'^{-1} = \sqrt{\Lambda}^{-1} Q^T \in \mathbb{R}^{k \times k}$

Here $\sqrt{\Lambda}$ is the diagonal matrix of square roots of eigenvalues: $(\sqrt{\Lambda})_{ii} = \sqrt{\Lambda_{ii}}$ , and $\sqrt{\Lambda}^{-1}$ has entries $1 / \sqrt{\Lambda_{ii}}$ . Both are well-defined because the eigenvalues of $\tilde{X}\tilde{X}^T$ are non-negative. In practice, very small eigenvalues (close to numerical zero) can cause instability in the inversion; the paper does not discuss thresholding or regularization, implying that the activation covariances encountered in these experiments are sufficiently well-conditioned. If an eigenvalue were exactly zero, the corresponding eigenvector direction would have zero activation variance and its contribution to the projection would be zero anyway.

4. Project the compression error: $\Delta W' = \Delta W Q' \in \mathbb{R}^{d \times k}$

This is a right-multiplication of a $d \times k$ matrix by a $k \times k$ matrix, costing $O(d k^2)$ operations. For a typical layer with $d = k = 4096$ , this is about $6.9 \times 10^{10}$ operations — very fast on GPU.

5. Rank- $r$ SVD on the projected error: $B' A' = U' \Sigma' V'^T = \text{SVD}_r(\Delta W')$ where $\text{SVD}_r$ denotes truncated SVD keeping only the top $r$ singular values and vectors. $U' \in \mathbb{R}^{d \times r}$ , $\Sigma' \in \mathbb{R}^{r \times r}$ , $V' \in \mathbb{R}^{k \times r}$ . Set $B' = U' \Sigma' \in \mathbb{R}^{d \times r}$ and $A' = V'^T \in \mathbb{R}^{r \times k}$ .

This is the dominant computational cost: truncated SVD of a $d \times k$ matrix, keeping only $r$ components. For $d = 4096$ , $k = 4096$ , $r = 128$ , randomized SVD algorithms (e.g., torch.svd_lowrank) can compute this in $O(d k r)$ time — about $2.1 \times 10^9$ operations, very manageable.

6. Project the right singular vectors back to original space: $A = A' Q'^{-1} = V'^T \sqrt{\Lambda}^{-1} Q^T \in \mathbb{R}^{r \times k}$

This is a $(r \times k) \times (k \times k)$ multiplication, but structured: $Q^T$ is orthogonal, and $\sqrt{\Lambda}^{-1}$ is diagonal scaling, so this can be computed as: first compute $V'^T Q^T$ (an $r \times k$ times $k \times k$ multiplication, $O(r k^2)$ ), then scale columns by $1/\sqrt{\Lambda_{ii}}$ (element-wise, $O(r k)$ ). Since $r \ll k$ (typical $r = 128$ , $k \approx 4096$ – $14336$ ), the $O(r k^2)$ term dominates and is comparable to the SVD cost.

7. Inference forward pass: $\text{output} = \hat{W} X + B' A X$

At inference time, the compressed weight $\hat{W}$ is applied as usual (potentially using a fused quantized kernel for low-bit weights, or a structured sparsity kernel for 2:4 pruned weights), and the low-rank residual $B'(A X)$ is computed in two steps: first $A X$ (an $r \times d$ matrix-vector multiply per token), then $B' (A X)$ (a $d \times r$ matrix-vector multiply). The total additional FLOPs per token is $2 r (d + k)$ — for $r = 128$ , $d = k = 4096$ , this is about $2.1 \times 10^6$ additional operations per token per layer, compared to $d \times k = 1.7 \times 10^7$ for the main linear layer, so roughly a 12% overhead. The paper's kernel optimization (Section 4.5) addresses the memory bandwidth bottleneck that makes this overhead larger in practice than in FLOP counting.

The Kernel Optimization: Fusing Low-Rank and Quantization Operations

Section 4.5 and Appendix A.6 describe a practical implementation challenge and its solution. The theoretical FLOP overhead of EoRA is modest, but in practice, adding a low-rank residual path increases inference latency more than expected because of data movement costs.

The problem: In a standard implementation, the compressed layer computes $\hat{W}X$ and writes the output to memory (L2 cache or DRAM). Then the low-rank path reads $X$ from memory, computes $A X$ (intermediate result), reads that result, computes $B' (A X)$ , and writes the final output to memory, which then must be added to the stored $\hat{W}X$ output. This doubles the number of memory reads and writes for the layer output, shifting the bottleneck from compute to memory bandwidth.

The solution: The paper proposes fusing the multiplication by $B'$ (the second step of the low-rank path) with the quantized matrix-vector product kernel for $\hat{W}$ . Specifically, the shared output buffer for the layer is pre-allocated, and both the quantized $\hat{W}X$ computation and the $B'(AX)$ computation write their results directly into this same buffer (with accumulation), eliminating the need to write $\hat{W}X$ to memory, read it back, and add $B'(AX)$ . The kernel is described: "we build our custom EoRA kernel on top of GPTQ's low-bit quantized matrix vector product kernel, pre-allocating the shared output prior to matrix vector multiplication and integrating the full-precision matrix vector multiplication of $B'$ into the quantized kernel reducing redundant memory access" (Appendix A.6).

The impact: Table 9 shows that for LLaMA3-70B, 3-bit quantization without EoRA provides a 1.7× speedup over FP16 (60ms → 35ms). With EoRA rank 128 and without the custom kernel, the speedup drops to 1.1× (54ms). With the custom kernel, it recovers to 1.4× (43ms). The kernel reduces latency by 11ms (roughly 20%) compared to the naive implementation. For 4-bit quantization, the pattern is similar: the kernel provides an additional ~0.3× speedup factor.

Practical Considerations: Calibration Data, Rank, and Quantization Robustness

Calibration data construction (Section 4.1): The paper uses task-specific calibration data to capture the activation distribution for the target task. The calibration set is constructed as:

Language generation (WikiText2): 128 concatenated sentences of length 2048 from the WikiText2 training set.
Commonsense reasoning (ARC-C): 32 concatenated sentences of length 2048 from the ARC training set, combined with 32 concatenated sentences from C4 (a general web-text corpus). The C4 portion is included to prevent overfitting to the specific ARC distribution.
Math reasoning (MathQA/GSM8K): 32 concatenated sentences of length 2048 from the MathQA or GSM8K training set, combined with 32 concatenated sentences from C4.

The total calibration sequence length is 2048 × (number of sentences). For the reasoning tasks with 64 total sentences, the total token count is approximately 131,072 tokens. This is a key practical advantage: EoRA needs only tens of thousands of tokens per task, making it applicable even when task-specific data is scarce. The ablation in Table 8 shows that EoRA maintains strong performance with as few as 32 calibration sentences (roughly 65,536 tokens), and saturates around 128–256 sentences — further calibration data beyond that point yields diminishing returns.

Rank selection (Section 4.3, Figure 2): EoRA is evaluated at ranks {64, 128, 256, 512}. The rank controls the tradeoff between compensation capacity and inference overhead: higher rank means more parameters in the low-rank matrices ( $B'$ has $d \times r$ elements, $A$ has $r \times k$ elements, so total parameter count per layer is $r(d + k)$ ) and more FLOPs at inference, but also higher accuracy. Figure 2 shows that EoRA's advantage over baselines grows with rank — at rank 512 on GSM8K, EoRA achieves 23.28% (vs. 8.64% for Act-S and 6.75% for ZeroQuant-V2), a gap of 14.64 percentage points.

Quantization of EoRA itself (Section 4.5, Table 13, Figure 3b): The low-rank matrices $B'$ and $A$ can themselves be quantized to further reduce memory overhead. The paper evaluates quantizing EoRA matrices to 4-bit and 3-bit precision. Key findings:

Quantizing EoRA from FP16 to 4-bit reduces total model size by up to 16.49% (for rank 512 on 2:4 pruned LLaMA3-8B: 11.70 GB → 9.77 GB) while incurring negligible accuracy loss (41.89% → 41.46% on ARC-C, a drop of 0.43%).
Quantizing to 3-bit is more aggressive; the accuracy drop becomes more noticeable (41.89% → 40.35% on ARC-C, a drop of 1.54%) but still leaves accuracy substantially higher than the uncompensated baseline (30.11%).
In some cases, quantizing EoRA improves accuracy slightly, which the paper attributes to "quantization acting as a form of regularization, as discussed in OFQ" (Section 4.5).

EoRA as Initialization for LoRA Fine-tuning (Section 4.4)

Although EoRA is fine-tuning-free, the paper also demonstrates that its low-rank matrices serve as excellent initialization for subsequent LoRA fine-tuning when additional computational resources are available. The procedure: use EoRA's $B'$ and $A$ matrices as the initial values for LoRA's low-rank adapters, then fine-tune only these adapters (keeping both the compressed backbone and the original weights frozen) using standard gradient-based training. Table 3 and Table 11 show that EoRA-initialized LoRA consistently outperforms both standard LoRA initialization (QLoRA) and SVD-based initialization (LoftQ). For 4-bit quantized LLaMA3-8B on MathQA, EoRA initialization achieves 56.04% vs. 53.96% for LoftQ and 51.42% for QLoRA — a 2.08 percentage point margin over the nearest competitor. The advantage is larger for more aggressive compression: on 3-bit quantized LLaMA3-8B, EoRA initialization achieves 53.90% vs. 48.17% for LoftQ (a 5.73 percentage point gap), and on 2:4 pruned LLaMA3-8B, 54.67% vs. 48.77% (a 5.90 percentage point gap). This is consistent with the pattern observed in the fine-tuning-free setting: EoRA's advantage over baselines grows with compression aggressiveness, because the larger the compression error, the more important it is to allocate the low-rank capacity to the right error components.

Design Choices and Their Justifications Summary

Eigendecomposition rather than Cholesky or other factorization of $X X^T$ : Eigendecomposition always exists for symmetric matrices (which $X X^T$ is by construction), while Cholesky decomposition (used by SVD-LLM) requires positive-definiteness, which can fail when the activation covariance has zero or near-zero eigenvalues — a common occurrence in high-dimensional transformer activations. This makes EoRA more numerically robust.
Projection by $Q\sqrt{\Lambda}$ rather than by $Q$ alone: The $\sqrt{\Lambda}$ scaling embeds eigenvalue importance into the projected error so that the subsequent unweighted SVD automatically prioritizes high-variance directions. Using $Q$ alone would merely rotate the error into the eigenbasis without reweighting, which would not change what SVD optimizes — the Eckart-Young theorem would still find the best rank- $r$ approximation in the rotated space, but this would correspond to equally-weighted error in the original space, defeating the purpose.
Right-multiplication of $\Delta W$ by $Q'$ rather than left-multiplication: The error $\Delta W$ has shape $d \times k$ (output × input). The activation covariance $X X^T$ has shape $k \times k$ (input × input). Right-multiplication by $Q'$ (which has shape $k \times k$ ) acts on the input dimension, which is exactly where the activation-based importance weights should be applied — different input channels have different importance. Left-multiplication would weight output dimensions differently, which is not what the activation covariance encodes.
Using the average activation $\tilde{X}$ rather than the full activation tensor: Computing $\tilde{X}$ as the mean over calibration tokens reduces the construction of $X X^T$ to a $k \times k$ eigendecomposition regardless of the number of calibration tokens. If the full $n \times k$ activation matrix were used, the SVD would be more complex. The paper does not discuss the effect of averaging on the eigenstructure, but since the covariance of the mean is proportional to the covariance of the individual samples, the eigendirections are preserved.
Single-step closed-form optimization rather than iterative updates (unlike LRC, CALDERA): Solving for $B'$ and $A'$ in one shot via SVD avoids the sequential error accumulation that can occur in alternating optimization, and ensures that the solution is globally optimal for the projected objective. This also makes EoRA compatible with the shared-backbone paradigm: since only the low-rank matrices (not the compressed weights) are modified, multiple task-specific EoRA adapters can coexist and be swapped dynamically.
Not modifying the compressed weights $\hat{W}$ : This is the constraint that distinguishes customized compensation from compression-aware fine-tuning. It is what enables multi-adapter deployment (one backbone, many task-specific adapters) and simplifies the workflow (no risk of degrading the compressed model's general capabilities through task-specific weight modification).

4. Key Insights and Innovations

Innovation 1: Difficulty-Conditioned Compute-Optimal Test-Time Scaling

EoRA's most fundamental contribution is not a new compression method, but rather the introduction of a new problem formulation — customized compensation — that reframes the relationship between model compression and task-specific accuracy as a decoupled, plug-and-play optimization rather than an entangled, one-shot decision.

Prior work in post-training compression operates under an implicit assumption: once a model is compressed, the accuracy loss is fixed. The only levers available are to choose a different compression format (e.g., 4-bit instead of 3-bit, or dense instead of 2:4 sparse) or to fine-tune the compressed model — both of which are heavyweight operations that produce a single, monolithic model. The paper's diagnosis is that this rigidity is an artifact of the compression paradigm, not an intrinsic property of compressed models. By separating the general compression (which produces a shared backbone) from the task-specific compensation (which produces lightweight adapters), EoRA introduces a new degree of freedom: the accuracy-latency tradeoff can be adjusted per task, per user, and even dynamically at inference time, without any modification to the compressed backbone.

This reframing is significant beyond the specific algorithm because it unlocks a deployment architecture that was previously impractical: a single compressed model served by a shared inference engine, with task-specific low-rank adapters loaded on demand (as supported by multi-adapter frameworks like vLLM). The paper does not merely propose a better error approximation method — it defines a problem setting where the separability of compression and compensation is the design objective, and then builds an algorithm that satisfies that objective. This is a conceptual shift from "how do we compress better?" to "how do we make compression flexible after the fact?"

The evidence that this reframing matters practically — not just conceptually — comes from the quantized-EoRA results (Figure 3b, Table 13). When EoRA adapters themselves are quantized to 4-bit, the total model size increase for a rank-128 adapter on a 3-bit quantized LLaMA3-8B is only 3% (4.63 GB → 4.78 GB), while recovering 10.6 percentage points of ARC-Challenge accuracy (20.90% → 31.48%). This means the cost of flexibility — the overhead of supporting the adapter infrastructure — is negligible compared to the accuracy gains it enables. If the adapters cost 20–30% extra memory, the flexibility argument would be weaker; the empirical finding that they cost 2–7% makes the case compelling.

Innovation 2: Activation Covariance as a Principal Weighting Function for Compression Error Approximation

The paper's core technical insight is that the activation covariance matrix — specifically, its eigenspace — provides a principled, task-specific importance function that transforms the unweighted error approximation problem (minimize $\|\Delta W - BA\|_F$ ) into the task-aware error approximation problem (minimize $\|\Delta W X - BA X\|_F$ ) without requiring iterative optimization. This is not a heuristic scaling trick; it is a theoretical equivalence derived by algebraic manipulation of the layer-wise compression loss.

The dominant approach in prior fine-tuning-free low-rank compensation is to scale compression errors by per-channel activation magnitudes — the intuition being that weight errors affecting high-magnitude input channels matter more. This is what ZeroQuant-V2, ASVD, LQER, SLiM, OATS, and QERA all do, with variations in exactly how the scaling is computed (mean absolute value, root-mean-square, max value, etc.). These methods share a common limitation: they only use the diagonal of the activation covariance matrix, discarding all off-diagonal correlation information. In a transformer layer, especially after self-attention operations that mix information across input dimensions, activation channels are correlated — sometimes strongly — and these correlations encode task-specific patterns that per-channel magnitude scaling cannot capture.

EoRA's eigendecomposition of $\tilde{X}\tilde{X}^T$ captures the full covariance structure: the eigenvectors represent the principal directions of activation variation (which may be linear combinations of many original input channels), and the eigenvalues quantify the importance of each direction. By projecting the compression error into this eigenspace before applying SVD, EoRA ensures that the rank budget is allocated to the error components that matter most for the task — not just the ones with the largest raw magnitudes, but the ones aligned with the dominant activation patterns in the calibration data. Theorem 1 formalizes this: the equivalence between $\|\Delta W' - B'A'\|_F$ and the true layer-wise compression loss is exact, not approximate, and it holds regardless of the correlation structure in $X X^T$ .

The empirical evidence for this being a fundamental advance rather than an incremental refinement comes from the systematic comparison in Table 18 (Appendix A.13). Against seven activation-statistics-based baselines — including the latest methods LQER, QERA, SLiM, and OATS — EoRA consistently outperforms all of them on both 4-bit and 3-bit quantized LLaMA3-8B. On 3-bit MathQA, EoRA achieves 29.11% vs. 25.60% for the best activation-scaling method (LQER), a 3.51 percentage point gap. This is not a marginal improvement; it suggests that the off-diagonal covariance structure — which all activation-scaling methods ignore — contains substantial task-relevant information that EoRA is uniquely able to exploit among fine-tuning-free methods.

This insight also clarifies why calibration data matters for fine-tuning-free compensation. The naive SVD approach (ZeroQuant-V2) uses zero calibration data — its approximation is task-agnostic. Activation-magnitude scaling methods use calibration data only to compute per-channel statistics — a first-order summary. EoRA uses calibration data to compute the full second-order covariance structure — a richer representation that captures how channels co-vary on the task. The progression from no calibration data → first-order statistics → second-order structure maps directly to the observed performance progression from ZeroQuant-V2 → Act-S → EoRA, visible consistently across Tables 1, 2, 4, and 5.

Innovation 3: The Closed-Form Solution Enables a Practical Speed-Accuracy-Compute Frontier Beyond Gradient-Based Methods

EoRA demonstrates that task-specific error compensation can be solved in closed form — via eigendecomposition followed by SVD — achieving accuracy competitive with or exceeding gradient-based methods while reducing the time cost from hours to minutes. This is not obvious a priori. The natural assumption (embodied by ApiQ, LoRA, and the iterative methods like CALDERA and LRC) is that minimizing the layer-wise compression loss with respect to low-rank matrices requires gradient-based optimization — that the objective is non-convex in the product $BA$ and therefore local iterative search is necessary. EoRA-side observation: the reparameterization via eigenspace projection makes the objective convex in the projected matrices $B'$ and $A'$ individually, and the global optimum has a closed-form expression via SVD.

The significance of this closed-form property extends beyond mere speed. It means EoRA's solution is deterministic and reproducible — given the same compressed model, calibration data, and rank, the result is always identical. Gradient-based methods (ApiQ, LoRA) depend on random initialization, learning rate schedules, batch ordering, and early stopping criteria, all of which introduce variance and require hyperparameter tuning. For a practitioner who needs to deploy adapters for dozens of tasks, the elimination of this tuning burden is a practical advantage of the same order as the runtime reduction.

The evidence for the speed-accuracy tradeoff advantage is in the comparison with ApiQ (Section 4.2.1): on 2:4 pruned LLaMA3-8B, EoRA achieves 34.64% ARC-C in ~15 minutes vs. ApiQ's 34.21% in ~2.5 hours, and 29.91% MathQA vs. ApiQ's 28.77%. EoRA is simultaneously faster and more accurate — a strict Pareto improvement. On GSM8K, ApiQ edges out EoRA (14.55% vs. 13.95%), but this 0.6 percentage point difference comes at a 10× time cost. For a user who values rapid adaptation above the last fraction of a percent, EoRA dominates.

This finding also has implications for what the field should consider "necessary" for error compensation. The success of a closed-form method at recovering the majority of the accuracy lost to compression suggests that the compensation problem is largely determined by second-order activation statistics and that gradient-based refinement provides diminishing returns. This is consistent with the pattern in Table 18, where iterative methods like CALDERA (which alternates between updating quantized weights and low-rank matrices) perform well but do not dramatically outperform EoRA's single-shot solution — the first-order approximation captures most of the achievable gain. If this pattern generalizes to other models and tasks, it would shift research priorities away from developing more sophisticated optimization procedures and toward better methods for estimating and leveraging activation statistics.

Innovation 4: The Combination of Eigenspace Projection and Frozen Backbone Creates a New Point in the Design Space Between PTQ and PEFT

EoRA occupies a previously empty point in the design space of post-training model adaptation: fine-tuning-free, task-specific, and backbone-preserving. Each existing class of methods satisfies at most two of these three properties:

Post-training compression (GPTQ, SparseGPT): fine-tuning-free and backbone-modifying (the weights change), but not task-specific (the compression uses generic calibration data).
Naive SVD compensation (ZeroQuant-V2): fine-tuning-free and backbone-preserving, but not task-specific (no calibration data used).
Activation-scaling compensation (Act-S, LQER, et al.): fine-tuning-free and backbone-preserving, and uses calibration data — but as shown, the task-specificity is limited to first-order statistics.
Parameter-efficient fine-tuning (LoRA, QLoRA): task-specific and backbone-preserving, but requires gradient-based training — not fine-tuning-free.
Iterative compensation (CALDERA, LRC): task-specific but modifies the backbone and requires iterative optimization — neither fine-tuning-free nor backbone-preserving.

The intellectual contribution is that the combination of eigenspace projection with SVD achieves all three properties simultaneously, and the paper's theoretical analysis (Theorem 1) explains why this combination works when individually each piece — SVD alone, or activation statistics alone — falls short. The projection makes the SVD-objective equivalent to the task-specific compression loss (solving the task-specificity problem), while remaining a single-pass matrix decomposition (preserving the fine-tuning-free property), and operating only on the residual path (preserving the backbone).

This is not merely taxonomic elegance. It has direct practical consequences demonstrated in the paper: the ability to serve a single compressed model with multiple task-specific adapters (Figure 1), the ability to quantize the adapters without re-quantizing the backbone (Section 4.5), and the ability to use the adapters as initialization for further fine-tuning if desired (Section 4.4). Each of these capabilities follows directly from the backbone-preserving property, which is itself enabled by the closed-form nature of the solution — if the compensation required modifying the backbone (as in CALDERA or LRC), multi-adapter serving would require storing and loading full model copies per task.

The evidence that this design point is practically useful — not just theoretically interesting — comes from the kernel optimization results (Section 4.5, Table 9). Without the backbone-preserving property, there would be no opportunity for the paper's custom fused kernel: the kernel integrates the $B'$ matrix-vector product with the quantized $\hat{W}$ kernel, which requires that $\hat{W}$ be fixed and known at kernel compilation time. If $\hat{W}$ were modified per task (as in iterative methods), this fusion would be impossible or would require per-task kernel compilation. The 1.4× speedup achieved by the fused kernel is therefore a downstream consequence of the design choice to keep the backbone frozen — a practical benefit that would not be available to methods that modify the compressed weights.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses WikiText2 for language generation (perplexity), ARC-Challenge (ARC-C) for commonsense reasoning, and MathQA and GSM8K for math reasoning. All are standard benchmarks accessed via the LM-Evaluation-Harness framework (Gao et al., 2024). For calibration, the paper constructs task-specific sets: 128 concatenated sentences of length 2048 from WikiText2 training set for language generation; 32 sentences from ARC training set + 32 sentences from C4 for commonsense reasoning; 32 sentences from MathQA/GSM8K training set + 32 sentences from C4 for math reasoning (Section 4.1). The C4 mixing for reasoning tasks prevents overfitting to the narrow task distribution. Additional evaluations on CNN/DailyMail summarization (ROUGE-Lsum) and MMLU appear in Appendix A.10, Tables 14–15.
Base model(s). Experiments use LLaMA2-7B, LLaMA2-13B, and LLaMA3-8B, with LLaMA3-70B used only for kernel speed benchmarking (Table 9). The authors argue these models span a representative range of scales and architectures, and LLaMA3-8B in particular serves as the primary testbed because it shows the largest accuracy degradation under aggressive compression — making it the most challenging case for compensation methods. All models are compressed post-training using SparseGPT for pruning and GPTQ for quantization, with channel-wise asymmetric quantization applied throughout. The calibration dataset for both compression methods follows the settings in Huang et al. (2024; reference [19]).
Metrics. The paper reports: (1) perplexity on WikiText2 (lower is better), computed via the standard causal language modeling loss; (2) accuracy (%) on ARC-Challenge, MathQA, and GSM8K, evaluated using the LM-Evaluation-Harness framework with exact-match answer grading; (3) ROUGE-Lsum for CNN/DailyMail summarization; (4) layer-wise output discrepancy (Table 17) measured as the Frobenius norm difference between the original and compensated layer outputs. For inference speed (Table 9), the metric is average per-token latency in milliseconds for 128-token generation at batch size 1.
Baselines. The paper compares against: (1) ZeroQuant-V2 (Yao et al., 2024) — naive SVD applied directly to the compression error ΔW, the simplest fine-tuning-free low-rank compensation; (2) Act-S — activation-statistics-based scaling adapted from ASVD (Yuan et al., 2023), where error columns are scaled by per-channel average absolute activation values before SVD and projected back; (3) ApiQ (Liao et al., 2024) — gradient-based optimization of low-rank matrices to minimize the layer-wise compression loss, with the layer-wise variant used for fair comparison since more resource-intensive variants "align more closely with PEFT methods" (Section 4.1); (4) For the fine-tuning experiments (Section 4.4): QLoRA (Dettmers et al., 2023) with standard random initialization and LoftQ (Li et al., 2024) with SVD-based initialization; (5) In the extended comparisons (Appendix A.13, Table 18): FWSVD, LQER, LRC, CALDERA, QERA, SLiM, and OATS (all cited in Section A.13). The uncompressed full-precision model and the uncompensated compressed model are reported as reference points, not as baselines to beat — the goal is to close the gap between these two.
Generation budget / compute accounting. Since EoRA is a post-training compensation method, not a test-time sampling strategy, "compute budget" has two distinct meanings: (1) Optimization time: how long the compensation algorithm takes to produce the low-rank matrices from calibration data — EoRA and Act-S complete in "a few minutes" (~15 minutes for EoRA on LLaMA3-8B with rank 128), while ApiQ requires "over hours" (~2.5 hours per task). (2) Inference overhead: the additional FLOPs and latency from the low-rank residual path, measured as per-token latency and model size increase. The paper reports inference speed with and without the custom fused kernel (Table 9), and model size in GB for different quantization configurations of the EoRA matrices (Table 13, Figure 3b). The rank r directly controls both: higher rank means more parameters (r(d + k) per layer), higher FLOPs (2r(d + k) per token), but higher accuracy.
Cross-validation / statistical protocol. The paper does not report cross-validation in the standard machine learning sense — there is no hyperparameter tuning on a validation split. Instead, EoRA is a deterministic algorithm: given the same compressed model, calibration data, and rank, the output is identical every time. The ablation on calibration size (Table 8) serves as a sensitivity analysis, showing that EoRA's performance stabilizes around 64–128 calibration samples and does not degrade substantially with fewer samples. The paper does not report confidence intervals or statistical significance tests on the accuracy numbers, which is a limitation — the test sets for ARC-C and MathQA are fixed, and with 500 (ARC-C) and test-set-sized (MathQA, GSM8K) evaluation instances, the standard error on reported accuracy differences of 1–2 percentage points could be non-trivial.

Main Quantitative Results

Sparsity Error Compensation (Table 1, Table 4)

The headline results for compensating 2:4 structured sparsity (SparseGPT) on LLaMA3-8B, with all methods at rank 128, appear in Table 1: EoRA achieves 34.64% ARC-C (vs. 30.11% uncompensated, a 4.53 percentage point recovery out of a 20.31 point gap), 29.91% MathQA (vs. 26.43%, a 3.48 point recovery), and 13.95% GSM8K (vs. 2.12%, an 11.83 point recovery). The uncompressed baseline is 50.42% ARC-C, 40.10% MathQA, 36.23% GSM8K.

The comparison against baselines shows EoRA outperforming both fine-tuning-free methods: on ARC-C, EoRA beats ZeroQuant-V2 by 2.65 percentage points (34.64% vs. 31.99%) and Act-S by 2.90 points (34.64% vs. 31.74%). On MathQA, the margins are 3.42 points over ZeroQuant-V2 (29.91% vs. 26.49%) and 3.18 points over Act-S (29.91% vs. 26.73%). On GSM8K, the margins are even larger: 10.99 points over ZeroQuant-V2 (13.95% vs. 2.96%) and 10.69 points over Act-S (13.95% vs. 3.26%). Against the gradient-based ApiQ, EoRA is competitive: +0.43 points on ARC-C (34.64% vs. 34.21%), +1.14 points on MathQA (29.91% vs. 28.77%), but −0.60 points on GSM8K (13.95% vs. 14.55%). Critically, EoRA achieves these numbers in ~15 minutes vs. ApiQ's ~2.5 hours (Section 4.1).

Table 4 in the appendix extends these results to LLaMA2-7B and LLaMA2-13B. The pattern is consistent: EoRA outperforms ZeroQuant-V2 and Act-S across all model sizes and tasks. For LLaMA2-13B on ARC-C, EoRA reaches 37.54% vs. 34.30% uncompensated (a 3.24 point recovery), compared to ZeroQuant-V2 at 33.61% and Act-S at 34.12%. On MathQA for the same model, EoRA achieves 27.53% (vs. 25.92% uncompensated), beating ZeroQuant-V2 by 2.41 points (27.53% vs. 25.12%). The EoRA–ApiQ comparison on LLaMA2 models mirrors LLaMA3: EoRA wins on ARC-C (37.54% vs. 36.68%) and MathQA (27.53% vs. 27.16%) for LLaMA2-13B, and ties on ARC-C for LLaMA2-7B (32.67% vs. 32.67%) while trailing on MathQA (25.59% vs. 26.36%). GSM8K results show ApiQ ahead on LLaMA2 models (+1.36 and +1.22 points on 7B and 13B respectively), consistent with ApiQ's slight advantage on this task observed for LLaMA3-8B.

The compatibility experiment with Wanda pruning (Table 7, Appendix A.4) demonstrates that EoRA's advantage is not specific to SparseGPT. On LLaMA3-8B pruned to 2:4 sparsity with Wanda, EoRA achieves 34.81% ARC-C (vs. 27.04% uncompensated, a 7.77 point recovery), outperforming ZeroQuant-V2 by 4.35 points (34.81% vs. 30.46%) and Act-S by 5.04 points (34.81% vs. 29.77%). On MathQA, EoRA reaches 30.05% vs. 25.09% uncompensated (+4.96 points), beating ZeroQuant-V2 by 3.89 points and Act-S by 3.32 points. Against ApiQ, EoRA wins on both ARC-C (34.81% vs. 31.91%, +2.90 points) and MathQA (30.05% vs. 29.61%, +0.44 points), while trailing on GSM8K (11.52% vs. 12.81%).

Quantization Error Compensation (Table 2, Table 5)

The headline results for compensating GPTQ quantization on LLaMA3-8B, with all methods at rank 128, appear in Table 2. Two compression levels are tested: 4-bit and 3-bit.

4-bit quantization causes moderate degradation: from 50.42% to 45.90% on ARC-C, 40.10% to 34.07% on MathQA, and 36.23% to 27.74% on GSM8K. EoRA recovers 47.44% ARC-C (+1.54 points over uncompensated, vs. ZeroQuant-V2 at 45.24%, Act-S at 47.86%, ApiQ at 46.58%). Notably, Act-S slightly outperforms EoRA on 4-bit ARC-C (47.86% vs. 47.44%), the only instance where a fine-tuning-free baseline beats EoRA on this task. On MathQA, EoRA achieves 37.21% (vs. 34.07% uncompensated), beating ZeroQuant-V2 by 0.70 points (37.21% vs. 36.51%), Act-S by 1.37 points (37.21% vs. 35.84%), and ApiQ by 1.03 points (37.21% vs. 36.18%). On GSM8K, EoRA reaches 30.70% (vs. 27.74% uncompensated), a 2.96 point improvement, but ZeroQuant-V2 slightly edges it out at 31.23% — the gap is 0.53 points.

3-bit quantization causes catastrophic degradation: 50.42% → 20.90% on ARC-C (a 29.52 point drop), 40.10% → 22.37% on MathQA (17.73 point drop), and 36.23% → 0.45% on GSM8K (35.78 point drop — the model's math reasoning is essentially destroyed). This is where EoRA's advantage is most pronounced. On ARC-C, EoRA recovers to 31.74% (a 10.84 point improvement over uncompensated), compared to ZeroQuant-V2 at 30.02%, Act-S at 31.28%, and ApiQ at 30.46% — EoRA leads by 0.46–1.72 points. On MathQA, EoRA reaches 29.11% (+6.74 points over uncompensated), beating ZeroQuant-V2 by 2.68 points (29.11% vs. 26.43%), Act-S by 3.69 points (29.11% vs. 25.42%), and ApiQ by 2.25 points (29.11% vs. 26.86%). On GSM8K, the recovery is dramatic: EoRA achieves 11.90% (vs. 0.45% uncompensated, an 11.45 point improvement), compared to ZeroQuant-V2 at 3.79%, Act-S at 4.09%, and ApiQ at 10.79% — EoRA leads the next best fine-tuning-free method by 7.81 points and edges out ApiQ by 1.11 points.

The appendix Table 5 extends these results to LLaMA2-7B and LLaMA2-13B. The pattern differs slightly by model size: for LLaMA2-7B at 3-bit, EoRA achieves 35.83% ARC-C — a 4.18 point recovery over the 31.65% uncompensated baseline, and the best among all methods (ZeroQuant-V2: 34.47%, Act-S: 32.67%, ApiQ: 33.70%). On MathQA for the same model and compression, EoRA reaches 25.79% (vs. 23.50% uncompensated), a 2.29 point recovery, but Act-S actually achieves the highest recovery at 25.02% (a 1.52 point recovery). On GSM8K for LLaMA2-7B at 3-bit, EoRA reaches 7.50% (from 0.38% uncompensated), with ApiQ at 7.13%, Act-S at 2.57%, and ZeroQuant-V2 at 2.04%. For LLaMA2-13B at 3-bit, EoRA consistently wins across all tasks: 39.50% ARC-C (from 37.28%, a 2.22 point recovery), 27.20% MathQA (from 26.26%, a 0.94 point recovery), and 15.08% GSM8K (from 4.62%, a 10.46 point recovery). The GSM8K recovery on the 13B model (15.08% vs. 4.62% uncompensated) is particularly striking — EoRA more than triples the compressed model's math reasoning accuracy, with the next best method (ApiQ) at 14.32%.

A notable cross-model comparison (called out in Appendix A.2): EoRA-enhanced 3-bit LLaMA2-13B achieves 15.08% GSM8K, which substantially outperforms the uncompensated 4-bit LLaMA2-7B at 9.93% GSM8K. This demonstrates that EoRA can make it more effective to deeply compress a larger model and compensate, rather than lightly compress a smaller model — a finding with direct deployment implications.

Combined Sparsity + Quantization Error Compensation (Table 6, Appendix A.3)

For models that undergo both pruning (2:4 sparsity via SparseGPT) and quantization (4-bit via GPTQ), the accuracy degradation is severe. On LLaMA3-8B, ARC-C drops from 50.42% to 18.34%, MathQA from 40.10% to 19.89%, and GSM8K from 36.23% to 0.00% — the model literally cannot solve any math word problems. Perplexity on WikiText2 explodes from 6.13 to 86.15.

EoRA substantially recovers performance: ARC-C improves to 31.22% (+12.88 points), MathQA to 29.58% (+9.69 points), GSM8K to 10.16% (+10.16 points from zero), and WikiText2 perplexity drops from 86.15 to 12.60. Compared to baselines on ARC-C: EoRA (31.22%) beats ZeroQuant-V2 (29.35%) by 1.87 points, Act-S (27.90%) by 3.32 points, and ApiQ (30.71%) by 0.51 points. On MathQA: EoRA (29.58%) beats ZeroQuant-V2 (26.86%) by 2.72 points, Act-S (25.59%) by 3.99 points, and ApiQ (28.74%) by 0.84 points. On GSM8K, ApiQ achieves 11.06% vs. EoRA's 10.16% — a 0.90 point advantage for ApiQ.

For LLaMA2 models (Table 6), the pattern is consistent but the absolute numbers are less catastrophic due to the larger models' greater robustness to compression. On LLaMA2-13B combined compression, EoRA achieves 35.06% ARC-C (from 33.10% uncompensated), beating ZeroQuant-V2 by 1.79 points (35.06% vs. 33.27%) and Act-S by 0.42 points (35.06% vs. 34.64%), but trailing ApiQ at 36.17% by 1.11 points. On MathQA for the same model, EoRA reaches 27.06% (from 24.75%), beating ZeroQuant-V2 by 1.77 points and Act-S by 0.97 points, and essentially tying ApiQ (27.06% vs. 26.96%). The overall trend: as compression becomes more aggressive, EoRA's advantage over fine-tuning-free baselines grows, while ApiQ's gradient-based optimization provides a slight edge on some tasks — but at a roughly 10× time cost.

Rank Scaling Analysis (Figure 2, Table 10)

Section 4.3 and Figure 2 examine how compensation accuracy scales with rank r in {64, 128, 256, 512} when compensating 2:4 pruned LLaMA3-8B.

The key observation: EoRA's advantage over fine-tuning-free baselines grows with rank. On GSM8K at rank 64: EoRA achieves 10.77% vs. Act-S at 3.34% — a 7.43 point gap. At rank 128: 13.95% vs. 3.26% — a 10.69 point gap. At rank 256: 17.06% vs. 5.16% — an 11.90 point gap. At rank 512: 23.28% vs. 8.64% — a 14.64 point gap. The absolute improvement from doubling rank (128 → 256) is 3.11 points for EoRA vs. 1.90 points for Act-S, suggesting EoRA extracts more value per rank increment because its allocation is more efficient. On MathQA, the pattern is similar but less dramatic: at rank 512, EoRA reaches 34.17% vs. Act-S at 29.65% (a 4.52 point gap) and ZeroQuant-V2 at 30.38% (a 3.79 point gap).

On ARC-C, the scaling is notably different: EoRA's accuracy rises from 33.10% at rank 64 to 41.89% at rank 512 — an 8.79 point improvement over the rank range. Act-S moves from 30.46% to 36.18% (a 5.72 point improvement), and ApiQ from 33.10% to 36.69% (a 3.59 point improvement). EoRA's slope is steeper than both baselines', meaning the gap widens with rank. At rank 64, EoRA ties ApiQ (33.10% vs. 33.10%); at rank 512, EoRA leads by 5.20 points (41.89% vs. 36.69%). Against the uncompensated baseline of 30.11%, rank-512 EoRA recovers more than half of the gap to the full-precision model (41.89% vs. 50.42%).

Comparing EoRA to ApiQ specifically (Figure 2): on ARC-C, EoRA overtakes ApiQ at rank 128 and the gap widens monotonically. On MathQA, EoRA leads at all ranks, with the gap growing from 0.70 points at rank 64 to 1.54 points at rank 512. On GSM8K, ApiQ leads at rank 64 (11.52% vs. 10.77%) and rank 128 (14.55% vs. 13.95%), but EoRA overtakes at rank 256 (17.06% vs. 15.85%) and extends the lead at rank 512 (23.28% vs. 20.77%). This crossing pattern suggests that ApiQ's gradient-based optimization is effective at lower ranks where the optimization landscape is better-conditioned, while EoRA's principled eigenspace allocation becomes more important as the rank increases and the allocation problem becomes higher-dimensional.

The appendix Table 10 confirms these trends for LLaMA2-7B and LLaMA2-13B. EoRA consistently outperforms ZeroQuant-V2 and Act-S across all ranks, with the gap widening at higher ranks on most (though not all) model-task combinations.

Calibration Data Ablation (Table 8)

Table 8 reports MathQA accuracy for EoRA compensating 3-bit and 4-bit quantized LLaMA3-8B, with calibration set sizes varying in {16, 32, 64, 128, 256, 512} sentences.

For 4-bit quantization: accuracy improves from 36.62% (16 samples) to 37.60% (256 samples), a modest 0.98 point gain, before declining slightly to 37.30% at 512 samples. For 3-bit quantization: accuracy improves from 26.33% (16 samples) to 30.40% (512 samples), a 4.07 point gain, but the improvement largely saturates after 128 samples (30.34%). The calibration time scales approximately linearly: 6.33 minutes for 16 samples, 11.33 for 128, 20.89 for 512.

The key practical takeaway: EoRA is robust to limited calibration data. Using only 32 samples (roughly 65,536 tokens), 3-bit accuracy reaches 27.57% — only 1.54 points below the maximum at 128 samples (29.11%) and 2.83 points below the maximum at 512 samples (30.40%). The paper's default setting of 64 samples for the reasoning tasks (Section 4.1) is positioned near the elbow of the saturation curve, where most of the available gain has been realized without excessive calibration time. A similar saturation pattern for 4-bit models confirms that EoRA does not require large amounts of task-specific data — a property that distinguishes it from gradient-based fine-tuning methods.

A subtle finding: for 2:4 pruned models, the paper notes that "using as few as 32 calibration samples to compensate for a 2:4 pruned model can even yield better accuracy improvements than using 64 samples" (Section 4.3). The exact numbers are not reported, but this suggests that for certain compression types and tasks, the calibration size optimum is task-dependent and more data is not universally better — a practically useful heuristic for users with very limited calibration data.

Inference Speed and Kernel Optimization (Table 9, Figure 3a)

Table 9 reports per-token latency for 128-token generation on LLaMA3-70B, the largest model tested, comparing FP16, quantized-only, and quantized + EoRA with and without the custom fused kernel. This is the only latency experiment; all other experiments are accuracy-only, so this serves as a proof-of-concept that the low-rank overhead can be mitigated.

The baseline FP16 latency is 60ms (1×). 3-bit quantization alone achieves 35ms (1.7× speedup). Adding EoRA with rank 128 without the custom kernel reduces speedup to 1.1× (54ms) — the low-rank path essentially eliminates most of the quantization speedup due to memory traffic overhead. With the custom kernel, speedup recovers to 1.4× (43ms) — reclaiming most of the lost advantage. At rank 256 without the kernel, speedup drops to 1× (58ms) — identical to FP16, making the compression pointless from a latency perspective. With the kernel, 1.3× (48ms) is recovered. For 4-bit quantization (38ms baseline, 1.6× speedup), the kernel provides a 0.3× speedup factor at all ranks, e.g., rank 128 drops from 1× (61ms) to 1.2× (51ms) with the kernel.

The important practical finding: the custom kernel makes EoRA viable for latency-sensitive deployments, particularly at lower ranks (64–128) where the speedup remains substantial (1.4× at 3-bit with rank 128). At rank 256, the 1.3× speedup for 3-bit is still worthwhile but the overhead is more noticeable. For 4-bit, the kernel helps but the absolute speedup is more modest (1.1–1.2×), partly because 4-bit quantization alone provides only 1.6× speedup over FP16, leaving less room for the low-rank overhead to be absorbed before dropping below baseline.

Quantizing EoRA for Memory Reduction (Figure 3b, Table 13)

Figure 3b and Table 13 evaluate the accuracy–size tradeoff when EoRA matrices themselves are quantized to 4-bit or 3-bit.

For 2:4 pruned LLaMA3-8B with EoRA rank 128: the uncompensated model is 9.12 GB at 30.11% ARC-C. Adding FP16 EoRA increases size to 9.77 GB (+7.1%) and accuracy to 34.64%. Quantizing EoRA to 4-bit reduces size to 9.28 GB (+1.8% over uncompressed) with essentially no accuracy loss (34.47%, a 0.17 point drop). Quantizing to 3-bit gives 9.24 GB (+1.3%) at 34.72% — actually slightly higher than full-precision EoRA (the paper attributes this to quantization as regularization, citing OFQ). The rank-512 case is more dramatic: FP16 EoRA adds 2.58 GB (11.70 GB total, +28.3%), while 4-bit quantization of EoRA shrinks this to 0.65 GB overhead (9.77 GB total, +7.1%) with only 0.43 points of accuracy loss (41.89% → 41.46%).

For 3-bit quantized LLaMA3-8B with EoRA rank 128: the uncompensated model is 4.63 GB at 20.90% ARC-C. FP16 EoRA adds 0.65 GB (5.28 GB total, +14.0%) for 31.74% accuracy. 4-bit quantized EoRA: 4.78 GB (+3.2%) at 31.48% (a 0.26 point drop). The pattern consistently shows that 4-bit EoRA achieves almost all the accuracy benefit of FP16 EoRA at a fraction of the memory cost. The paper recommends "users quantize EoRA to 4-bit, as this significantly reduces inference latency and model size with kernel support, without causing any noticeable drop in accuracy" (Section 4.5).

A deeper pattern visible across compression settings: as the compression becomes more aggressive (from 4-bit → 3-bit → 2:4+4-bit), the accuracy gain from EoRA increases, and concurrently, quantizing EoRA becomes more favorable because the EoRA matrices represent a larger fraction of the total model parameters (since the backbone itself is smaller). For 3-bit compressed LLaMA3-8B with EoRA rank 512: FP16 EoRA nearly doubles the model size (4.63 GB → 7.16 GB, +54.6% for 18.0 points of ARC-C gain), but 4-bit quantized EoRA provides most of the gain (19.1 points) at only 14.0% size increase (4.63 GB → 5.28 GB). This is the regime where EoRA is most practically deployable.

EoRA as LoRA Initialization (Table 3, Table 11, Table 12)

Section 4.4 evaluates EoRA's low-rank matrices as initialization for standard LoRA fine-tuning, compared against QLoRA (random initialization) and LoftQ (SVD-based initialization). All methods fine-tune only the low-rank adapters (rank 128 on every linear layer), keeping the compressed backbone frozen, for 3 epochs with batch size 64, learning rate 1e-5, and cosine scheduling. Fine-tuning is done on the ARC training set for ARC-C evaluation, and on the MathQA training set for math evaluation.

Table 3 reports the 4-bit quantization case: EoRA initialization achieves 55.46% ARC-C and 56.04% MathQA, compared to QLoRA at 54.09%/51.42% and LoftQ at 54.52%/53.96%. The margin over LoftQ is 0.94 points on ARC-C and 2.08 points on MathQA. The full-precision standard LoRA baseline reaches 56.39%/53.56%, meaning EoRA initialization enables compressed-model fine-tuning that is within 0.93 points on ARC-C and actually exceeds the full-precision baseline by 2.48 points on MathQA.

Table 11 extends this to more compression settings. For 2:4 sparsity: EoRA initialization achieves 48.54% ARC-C and 54.67% MathQA, compared to QLoRA (41.30%/45.42%) and LoftQ (43.68%/48.77%) — margins of 4.86 and 5.90 points over LoftQ. For 3-bit quantization: EoRA reaches 47.44%/53.90% vs. QLoRA (30.29%/34.10%) and LoftQ (44.70%/48.17%) — margins of 2.74 and 5.73 points. The pattern is clear: EoRA initialization provides larger gains over LoftQ as compression becomes more aggressive and the initialization quality matters more.

Table 12 ablates the effect of reduced fine-tuning data, evaluating 2:4 pruned LLaMA3-8B with 100%, 50%, and 30% of the training data. At 50% data: EoRA initialization achieves 46.41% ARC-C vs. LoftQ at 41.46% (a 4.95 point margin) and QLoRA at 38.56% (a 7.85 point margin). At 30% data: 43.85% vs. 39.76% vs. 36.77% — the margins shrink but remain substantial (4.09 and 7.08 points). This demonstrates that EoRA initialization is robust to data scarcity, consistent with EoRA itself requiring minimal calibration data.

Additional Task Results (Tables 14, 15)

Appendix A.10 extends evaluation beyond the main three reasoning tasks. On CNN/DailyMail summarization (Table 14), 4-bit quantized LLaMA3-8B achieves ROUGE-Lsum of 0.1672 uncompensated; EoRA improves this to 0.1812, outperforming ZeroQuant-V2 (0.1798), Act-S (0.1786), and ApiQ (0.1804). The gaps are small in absolute terms but consistent. For 3-bit quantization, EoRA lifts the score from 0.0650 to 0.1463 — more than doubling it — while the nearest competitor (ApiQ) reaches only 0.1357. On MMLU (Table 15), EoRA recovers 4-bit quantized LLaMA3.2-3B from 24.16% to 52.93% at rank 128 — nearly recovering the full-precision accuracy of 54.19%. Even rank 32 achieves 52.53%, demonstrating that significant recovery is possible with very low rank on this benchmark.

Layer-wise Discrepancy Analysis (Table 17)

Table 17 in Appendix A.12 measures the per-layer output discrepancy (Frobenius norm of activation difference from the full-precision model) for the q-projector layers of 4-bit quantized LLaMA2-7B, sampled at layers {0, 5, 10, 15, 20, 25, 30}. This is a diagnostic experiment rather than an end-task evaluation — it measures whether EoRA is actually reducing the per-layer error that its objective function targets.

GPTQ alone produces discrepancies that grow in deeper layers: 2.3 at layer 0, peaking at 13.4 at layer 10, then settling around 10.7–11.5 in later layers. EoRA reduces these substantially: at layer 20, from 10.7 (GPTQ) to 5.8 (EoRA); at layer 30, from 11.5 to 5.4. The reduction is roughly 40–50% across layers. Against baselines: EoRA matches or beats ApiQ at all layers (e.g., layer 20: 5.8 vs. 6.2; layer 30: 5.4 vs. 5.1) while being dramatically faster. Against ZeroQuant-V2 and Act-S, the advantage is larger: at layer 20, EoRA's 5.8 compares to 9.7 (ZeroQuant-V2) and 8.4 (Act-S). This per-layer diagnostic confirms that EoRA's theoretical guarantee — minimizing the layer-wise compression loss — translates to measurable improvements in layer-wise activation fidelity.

Extended Comparison with Recent Low-Rank Methods (Table 18)

Table 18 in Appendix A.13 provides the paper's most comprehensive baseline comparison, evaluating EoRA against nine recent low-rank methods on 4-bit and 3-bit quantized LLaMA3-8B using MathQA.

On 4-bit MathQA: EoRA achieves 37.21%, the highest among all methods. The next best is CALDERA at 36.70%, followed by LRC at 36.40%, ZeroQuant-V2 at 36.51%, and ApiQ at 36.18%. The activation-statistics-based methods (LQER: 35.46%, SLiM: 35.90%, QERA: 35.90%, OATS: 36.01%) cluster together roughly 1–1.5 points below EoRA. FWSVD achieves 35.64%, and Act-S achieves 35.84%.

On 3-bit MathQA: EoRA achieves 29.11%, again highest. LRC reaches 28.64% (0.47 points behind), CALDERA at 28.10% (1.01 points behind), ApiQ at 26.86% (2.25 points behind). The activation-statistics methods range from 25.30% (OATS) to 26.30% (FWSVD) — a cluster roughly 2.5–3.8 points below EoRA. The gap between EoRA and the best activation-statistics-based method (FWSVD at 26.30%) is 2.81 points on 3-bit, substantially larger than the 1.57 point gap on 4-bit — consistent with the pattern that EoRA's advantage grows with compression aggressiveness.

The comparison is notable because it includes the latest fine-tuning-free methods (LQER, QERA, SLiM, OATS), iterative methods (LRC, CALDERA), and the training-based ApiQ, and EoRA ranks first across both compression levels. The margins over the next-best method are modest (0.51 and 0.47 points on 4-bit and 3-bit respectively), suggesting that while EoRA is consistently the best, several recent methods are approaching similar performance. The real differentiation remains the combination of accuracy, speed (minutes vs. hours for iterative methods), and backbone preservation (vs. LRC and CALDERA which modify compressed weights).

2-Bit Quantization Results (Table 19)

Table 19 in Appendix A.14 evaluates EoRA in the extreme 2-bit regime, where RILQ (Lee et al., 2025) — a gradient-based method using cross-entropy loss and output distillation — is the state-of-the-art. On 2-bit GPTQ-quantized LLaMA3-8B MathQA: the uncompensated model achieves 18.22%. LoftQ (SVD initialization + RILQ fine-tuning) reaches 35.80%. EoRA alone (no fine-tuning) reaches 36.89% — outperforming LoftQ + RILQ fine-tuning by 1.09 points, despite being fine-tuning-free. RILQ with its standard initialization achieves 37.60%. Combining EoRA initialization with RILQ fine-tuning reaches 38.90% — the best result, outperforming RILQ alone by 1.30 points.

This experiment demonstrates two things: (1) even in the extreme 2-bit regime where gradient-based methods are expected to dominate, EoRA's closed-form initialization is competitive with full fine-tuning; (2) EoRA and gradient-based methods are complementary — the combination outperforms either alone, mirroring the pattern observed in Section 4.4 where EoRA initialization boosts LoRA fine-tuning.

Ablation Studies and Robustness Checks

Rank variation (ranks 64, 128, 256, 512): EoRA's advantage over fine-tuning-free baselines (ZeroQuant-V2, Act-S) widens monotonically with rank on GSM8K (gap grows from 7.43 to 14.64 points) and MathQA, and reaches 5.20 points over ApiQ at rank 512 on ARC-C for 2:4 pruned LLaMA3-8B (Figure 2, Table 10). The gains do not saturate at rank 512, suggesting higher ranks would yield further improvement — a finding with practical implications for users with generous memory budgets.
Calibration size (16, 32, 64, 128, 256, 512 samples): EoRA saturates around 128 samples for 3-bit quantized LLaMA3-8B MathQA (Table 8), with 32 samples achieving 27.57% vs. 29.11% at 64 samples and 30.34% at 128 samples. The finding that 32 samples can outperform 64 for pruned models (Section 4.3) indicates a non-monotonic relationship between calibration size and accuracy that warrants further investigation.
Compression method compatibility (SparseGPT vs. Wanda): EoRA generalizes across pruning algorithms: on Wanda-pruned 2:4 LLaMA3-8B, EoRA outperforms ZeroQuant-V2 by 4.35–5.04 points and Act-S by 3.32–10.01 points across ARC-C, MathQA, and GSM8K (Table 7). The absolute recoveries are larger for Wanda than SparseGPT (e.g., +7.77 vs. +4.53 on ARC-C), likely because Wanda's activation-based pruning creates different error patterns that EoRA's eigenspace projection handles well.
Group-wise vs. channel-wise quantization: EoRA maintains its advantage under group-wise quantization with group size 128 (Table 16): on 4-bit MathQA, EoRA achieves 39.16% (matching the full-precision 40.10% within 0.94 points), vs. ZeroQuant-V2 at 38.92% and Act-S at 38.49%. On 3-bit, EoRA reaches 35.10% vs. LRC at 34.80% and ApiQ at 34.80%. The paper originally uses channel-wise quantization consistent with GPTQ defaults; this ablation confirms robustness to the more common group-wise setting.
EoRA quantization (FP16 → 4-bit → 3-bit): EoRA matrices are robust to post-hoc quantization: 4-bit quantization of EoRA reduces memory overhead dramatically (e.g., +28.3% → +7.1% for rank 512 on 2:4 pruned LLaMA3-8B) with negligible accuracy loss (41.89% → 41.46%, a 0.43 point drop, Table 13). 3-bit quantization incurs larger losses (41.89% → 40.35%, a 1.54 point drop) but still substantially outperforms the uncompensated baseline (30.11%). Some cases show improved accuracy with quantization (e.g., rank 128 on 2:4 pruned: 34.64% → 34.72% at 3-bit), attributed to quantization as regularization (Section 4.5).
Fine-tuning data ratio (100%, 50%, 30%): EoRA-initialized LoRA fine-tuning is robust to reduced training data on 2:4 pruned LLaMA3-8B (Table 12). At 30% data, EoRA initialization achieves 4.09 points over LoftQ and 7.08 points over QLoRA on ARC-C, and the margins are comparable to those at 100% data (4.86 and 7.24 points). This robustness matches EoRA's own calibration-data efficiency.
EoRA alone vs. EoRA + fine-tuning on 2-bit quantization: On 2-bit GPTQ LLaMA3-8B MathQA (Table 19), EoRA alone (36.89%) slightly outperforms LoftQ + RILQ fine-tuning (35.80%) despite being fine-tuning-free. The combination EoRA + RILQ (38.90%) outperforms RILQ alone (37.60%) by 1.30 points. This demonstrates complementarity with state-of-the-art 2-bit methods.
Layer-wise output discrepancy: EoRA reduces per-layer activation error by 40–50% across layers of 4-bit quantized LLaMA2-7B (Table 17), matching or beating ApiQ's discrepancy reduction at a fraction of the optimization time. The reduction is uniform across early and late layers, showing the method does not overfit to specific layer types.
Task generalization (summarization, MMLU): EoRA's gains extend beyond the main three reasoning benchmarks to CNN/DailyMail summarization (3-bit ROUGE-Lsum: 0.0650 → 0.1463, Table 14) and MMLU (4-bit LLaMA3.2-3B: 24.16% → 52.93% at rank 128, Table 15), demonstrating the approach is not tuned to a specific task format.

Critical Assessment

The experimental section is thorough along the dimensions it explores, but several claims warrant scrutiny against what was actually tested.

Does the evidence support EoRA as a "fine-tuning-free" method that "swiftly improves task-specific accuracy"? Yes, with qualifications on "swiftly." The experiments convincingly show that EoRA completes in ~15 minutes for LLaMA3-8B with rank 128 (Section 4.1, Table 8 calibration times), compared to ~2.5 hours for ApiQ. This is a genuine order-of-magnitude speedup over the nearest gradient-based baseline. However, the paper does not report the eigendecomposition time as a function of model scale. For LLaMA3-8B, the largest intermediate FFN dimension is 14,336, so the eigendecomposition of a 14,336 × 14,336 matrix costs roughly O(14,336³) ≈ 3 × 10¹² operations — manageable on an H100. For a 70B model (used only for latency benchmarking), this cost would grow substantially (LLaMA3-70B's intermediate dimension is 28,672, giving O(2.4 × 10¹³) operations per FFN layer). The paper does not report EoRA optimization time for the 70B model — only inference latency. Whether EoRA remains "a few minutes" at 70B scale is unverified.

Does the evidence support that eigenspace projection (capturing full covariance) outperforms per-channel activation scaling (diagonal only)? The experiments strongly support this. Across every main comparison — sparsity compensation (Tables 1, 4, 7), quantization compensation (Tables 2, 5), combined compression (Table 6), rank scaling (Figure 2, Table 10), and extended baselines (Table 18) — EoRA outperforms Act-S, often by substantial margins (e.g., 10.69 points on GSM8K at rank 128, Table 1; 3.69 points on 3-bit MathQA, Table 2). The extended comparison in Table 18 shows EoRA leading a family of activation-scaling methods (LQER, QERA, SLiM, OATS) that all share the diagonal-only limitation, with the gap widening under more aggressive compression. This is strong evidence that the off-diagonal covariance structure captured by eigendecomposition contains practically significant task-specific information.

However, the theoretical mechanism — that eigendecomposition captures correlations between activation channels — is never directly validated through an ablation that isolates the correlation effect. A natural experiment would be to compare EoRA against a method that uses the full eigenvalues but diagonalizes the covariance (i.e., uses $\Lambda$ for scaling but ignores the rotation $Q$ ). This would test whether the eigenvalue weighting alone (without the eigenbasis rotation) accounts for most of the gain, or whether the rotation into the decorrelated eigenbasis is essential. This ablation is absent, leaving the precise source of EoRA's advantage — eigenvalue weighting vs. decorrelation — unresolved.

Does the evidence support that EoRA "outperforms prior fine-tuning-free low-rank methods"? Yes, with a minor exception. Against the two primary fine-tuning-free baselines (ZeroQuant-V2 and Act-S), EoRA leads on nearly every task-model-compression combination. The exceptions are narrow: Act-S beats EoRA on 4-bit ARC-C by 0.42 points (47.86% vs. 47.44%, Table 2), and ZeroQuant-V2 beats EoRA on 4-bit GSM8K by 0.53 points (31.23% vs. 30.70%, Table 2). These are marginal differences, and EoRA's dominance elsewhere is substantial. Against the extended set of recent low-rank methods (Table 18), EoRA ranks first on both 4-bit and 3-bit MathQA, though the margins over LRC (0.51 points on 4-bit, 0.47 points on 3-bit) are modest enough that a different calibration data split could potentially change the ordering.

Does the evidence support that EoRA can "freely balance the trade-off between accuracy and computational overhead"? This claim requires examining the rank scaling results (Figure 2) and quantization results (Figure 3b, Table 13). The rank scaling curves show a smooth, monotonic improvement in accuracy with rank, providing users a dial: more rank = more accuracy at higher computational cost. The quantization results show that quantizing EoRA to 4-bit provides nearly all the accuracy at a fraction of the memory cost. Together, these demonstrate that EoRA indeed enables flexible accuracy-overhead tradeoffs. However, the paper does not provide a unified latency-vs-accuracy plot across all rank and quantization combinations — users must cross-reference Figure 2 (accuracy vs. rank) with Table 9 (latency vs. rank) and Table 13 (size vs. quantization) to construct the full tradeoff space. A Pareto frontier plot combining all three axes would have strengthened this claim considerably.

Does the evidence support that EoRA can serve as a superior LoRA initialization? The evidence in Tables 3, 11, and 12 is consistent and convincing. EoRA initialization outperforms both QLoRA and LoftQ across all compression settings (2:4, 4-bit, 3-bit) and both tasks tested (ARC-C, MathQA). The advantages are substantial (2–6 points over LoftQ) and persist under reduced fine-tuning data (Table 12). However, the fine-tuning experiments are limited: only one learning rate (1e-5), one schedule (cosine), one epoch count (3), and one rank (128) are tested. It is possible that with hyperparameter tuning, QLoRA or LoftQ could close the gap — the paper does not demonstrate that EoRA initialization is robustly superior across hyperparameter settings, only that it is superior at the specific settings used.

Does the paper adequately address the scalability of the eigendecomposition? No. The computational cost of eigendecomposition — O(k³) per layer for input dimension k — is mentioned in passing in Section A.13 ("this degenerates the iterative process" in the CALDERA discussion) but never quantified or benchmarked. For LLaMA3-8B, the largest k is 14,336 (intermediate FFN dimension), and there are 32 decoder layers with multiple linear projections each. The total optimization time of "~15 minutes" is reported as a single number without a breakdown of how much time is spent on eigendecomposition vs. SVD vs. matrix multiplications. For a 70B model, the optimization time could be substantially longer, and the paper does not report it. Users considering EoRA for larger models have no guidance on expected optimization cost.

Does the paper adequately compare against methods that modify the compressed backbone? The paper's stated motivation is the shared-backbone paradigm — EoRA preserves the compressed weights so one backbone can serve multiple tasks with task-specific adapters. The paper compares against iterative methods (LRC, CALDERA) on accuracy alone (Table 18) but never quantifies the practical benefit of the backbone-preserving property. There is no experiment showing, for example, storage savings from using one backbone + N adapters vs. N separate task-specific compressed models, or multi-task inference throughput with adapter swapping (despite mentioning vLLM's multi-adapter support in Figure 1). The backbone-preserving property is asserted as valuable but its practical benefits are not measured. This is a significant gap for a paper whose problem formulation centers on this property.

Missing experiments that would strengthen the paper:

Multi-adapter deployment demonstration: An experiment showing that EoRA adapters for multiple tasks can be served from a single compressed backbone with competitive throughput would directly validate the customized compensation problem formulation.
Eigendecomposition wall-clock time scaling: Timing breakdowns per layer and per model scale (7B, 13B, 70B) would allow practitioners to estimate EoRA optimization cost for their own models.
Eigenvalue-only ablation: An experiment comparing EoRA against a method that scales $\Delta W$ by $\sqrt{\Lambda}$ (eigenvalues) but does not rotate via $Q$ (eigenvectors) would isolate the contribution of decorrelation from eigenvalue weighting.
Statistical significance: With test sets of 500+ questions, standard errors on the reported accuracy differences are computable. Reporting confidence intervals or at minimum noting that differences below ~1–2 points may not be statistically significant would add rigor.
Non-transformer architecture: All experiments are on LLaMA-family decoder-only transformers. Whether the eigenspace approach works for encoder-decoder models or non-text modalities is unexplored.
Latency–accuracy Pareto frontier: A unified plot across all rank and quantization combinations would directly support the "freely balance the trade-off" claim.
Comparison with simple LoRA fine-tuning on the same calibration data: The paper compares EoRA against fine-tuning-free baselines and against full dataset fine-tuning, but never asks: what if you spend the same ~15 minutes doing a few steps of gradient-based LoRA optimization on the same 64 calibration samples? This is the most direct head-to-head against "just do a tiny bit of fine-tuning" and its absence leaves open the question of whether the closed-form nature of EoRA provides benefits beyond compute efficiency — e.g., determinism, no hyperparameter tuning, guaranteed convergence.

Summary of evidential strength: The experiments strongly support EoRA's superiority over existing fine-tuning-free methods for the specific models (LLaMA2/3), compression techniques (SparseGPT, GPTQ, Wanda), and tasks (commonsense reasoning, math reasoning, language modeling, summarization) tested. The theoretical guarantee (Theorem 1) is backed by consistent empirical gains, and the key ablations (rank, calibration size, quantization) demonstrate robustness. The primary weaknesses are: (1) no direct validation of the practical benefit of backbone preservation (multi-adapter serving), (2) no eigendecomposition cost scaling analysis beyond 8B parameters, (3) missing the eigenvalue-only ablation that would isolate the mechanism of gain, and (4) no comparison with short gradient-based fine-tuning on the same calibration data. The paper's strongest empirical contribution — that eigenspace projection consistently beats per-channel scaling, especially under aggressive compression — is well-supported. The paper's broader deployment claims about flexibility and multi-task serving, while plausible, are not empirically validated in this study.

6. Limitations and Trade-offs

The Cost of Eigendecomposition Is Unmeasured and May Be Prohibitive at Scale

The assumption or constraint. EoRA's core computation — eigendecomposition of the activation covariance matrix $\tilde{X}\tilde{X}^T \in \mathbb{R}^{k \times k}$ for every compensated linear layer — has cubic complexity $O(k^3)$ in the layer's input dimension $k$ . The paper reports total optimization time as "a few minutes" (~15 minutes for LLaMA3-8B with rank 128; Section 4.1, Table 8), but never provides a breakdown of how much time is spent on eigendecomposition versus SVD versus matrix multiplications, nor does it measure or project this cost for larger models. The largest model for which optimization time is reported is LLaMA3-8B ( $k = 14{,}336$ for FFN intermediate layers). For LLaMA3-70B, used only for inference latency benchmarking (Table 9), the intermediate FFN dimension is $28{,}672$ , and the eigendecomposition cost per such layer grows by roughly $(28{,}672 / 14{,}336)^3 = 8\times$ compared to 8B. Across 80 layers with multiple linear projections each, the total eigendecomposition time for a 70B model could be on the order of hours rather than minutes.

The consequence. Practitioners considering EoRA for large-scale deployments (70B, 405B, or larger models) have no empirical guidance on the optimization time they should expect. The paper's headline claim of "a few minutes" applies only to the 7B–13B scale at which experiments were run. If optimization time for a 70B model stretches to hours, EoRA's practical advantage over gradient-based methods like ApiQ (which the paper reports requires "over hours" at 8B scale) may narrow or vanish — the speedup factor may not be constant across model scales.

What evidence exists in the paper. The paper acknowledges the complexity implicitly but provides no measurement. In the discussion of CALDERA (Section A.13), the paper notes that "the eigendecomposition of a $k \times k$ symmetric matrix has complexity $O(k^3)$ " but never quantifies wall-clock time at any scale. Table 8 provides calibration time as a function of calibration size for LLaMA3-8B (ranging from 6.33 minutes for 16 samples to 20.89 minutes for 512 samples), but these numbers conflate eigendecomposition, SVD, and all other operations into a single total. The 70B latency experiments (Table 9) measure only inference latency, not optimization time. The paper does not report optimization time for LLaMA2-13B either.

Mitigation status. Not addressed. The paper offers no analysis of how eigendecomposition cost scales with model dimension, no timing breakdown by operation, and no suggestions for reducing this cost (e.g., randomized eigendecomposition, layer subsampling, or incremental update strategies). For users with 70B+ models, this is a first-order practical question that the paper leaves unanswered.

The Backbone-Preserving Property's Practical Benefits Are Asserted but Never Measured

The assumption or constraint. The paper's problem formulation — customized compensation — centers on a specific constraint: "the weights of the compressed model are not modified during compensation. This enables deployment of a single, general compressed backbone alongside lightweight, task-specific low-rank modules that can be dynamically loaded as needed" (Section 1). This constraint is what distinguishes EoRA from iterative methods like CALDERA and LRC that modify compressed weights, and it is the justification for the shared-backbone deployment architecture illustrated in Figure 1. The paper invokes compatibility with multi-adapter inference frameworks like vLLM as a motivating use case.

The consequence. The practical value of backbone preservation — storage savings from one backbone + $N$ adapters versus $N$ separate task-specific compressed models, multi-task serving throughput, adapter swapping latency — is never quantified. A practitioner deciding whether to adopt EoRA over a method that modifies the backbone (e.g., fine-tuning or CALDERA) cannot assess whether the backbone-preserving constraint is worth accepting any accuracy disadvantage that comes with it, because the paper provides no measurements of the benefits. It is possible that storage savings are marginal (if adapters are a small fraction of model size, keeping $N$ full model copies may be acceptable) or that adapter swapping overhead in vLLM negates the deployment simplicity.

What evidence exists in the paper. None. Figure 1 depicts the multi-adapter deployment conceptually. Section 4.5 and Table 13 measure the storage overhead of EoRA adapters (e.g., rank-128 FP16 EoRA adds 0.65 GB to a 3-bit compressed 4.63 GB model, a 14% increase), but there is no comparison against the storage cost of maintaining separate task-specific compressed models. No experiment runs multi-task inference with adapter swapping to measure throughput or latency overhead of dynamic adapter loading. The vLLM integration is mentioned as future-facing infrastructure (Section 1: "enabling efficient integration with existing multi-adapter inference frameworks (e.g., vLLM)") but is never demonstrated.

Mitigation status. Not addressed. The paper treats backbone preservation as self-evidently valuable, but for a systems-oriented contribution that proposes a new deployment paradigm, the absence of any measurement of that paradigm's practical benefits is a significant gap. The authors do not acknowledge this as a limitation or suggest future work to validate the multi-adapter deployment scenario.

EoRA's Advantage Over Activation-Scaling Methods Is Not Fully Diagnosed — the Eigenvalue-Weighting vs. Decorrelation Distinction Is Unresolved

The assumption or constraint. EoRA's theoretical contribution (Theorem 1) is that projecting the compression error by $Q' = Q\sqrt{\Lambda}$ before SVD makes the approximation minimize the layer-wise compression loss. This projection does two things simultaneously: (1) it rotates the error into the eigenbasis via $Q$ (decorrelating the activation channels), and (2) it scales each eigen-direction by $\sqrt{\Lambda_{ii}}$ (weighting by eigenvalue importance). The paper compares EoRA against Act-S, which scales by per-channel activation magnitudes but does not rotate (effectively assuming the activation covariance is diagonal). EoRA consistently outperforms Act-S, but the experiment does not distinguish whether the gain comes from the eigenvalue weighting (better importance scores) or from the decorrelation (capturing off-diagonal covariance structure). In the paper's own framing (Section 3), the advantage of eigenspace projection over diagonal scaling is that "activation channels are highly correlated" and a diagonal scaling "discards all off-diagonal correlation information." But this causal claim is never isolated experimentally.

The consequence. If the majority of EoRA's gain comes from better per-channel importance weighting (using eigenvalues rather than mean absolute activations as importance scores), then simpler methods that use the diagonal of $\Lambda$ without the full eigendecomposition might achieve comparable performance at lower computational cost. Conversely, if the decorrelation via $Q$ is essential, then any diagonal method — regardless of how the scaling is computed — will asymptotically underperform EoRA as compression becomes more aggressive and the off-diagonal structure matters more. Without the ablation, users do not know which aspect of EoRA to prioritize if they need to approximate the method under tighter compute budgets.

What evidence exists in the paper. The paper demonstrates that EoRA outperforms a family of diagonal-scaling methods (Act-S, LQER, QERA, SLiM, OATS) in Table 18, with the gap widening under more aggressive compression (the 3-bit EoRA–LQER gap is 3.51 points on MathQA vs. 1.75 points at 4-bit). This is consistent with the hypothesis that off-diagonal covariance matters more when compression error is larger, but it does not isolate the mechanism. The paper provides no eigenvalue-only baseline (e.g., scaling $\Delta W$ by $\sqrt{\Lambda}$ without rotating by $Q$ , or equivalently, using only the diagonal of $Q \Lambda Q^T$ after eigendecomposition). The theoretical derivation in Theorem 1 treats the full $Q\sqrt{\Lambda}$ projection as a single operation; there is no decomposition of the proof into eigenvalue-weighting and decorrelation components that would suggest which matters more.

Mitigation status. Not addressed. The paper does not acknowledge this as an open question, nor does it suggest the eigenvalue-only ablation as future work. For a methods paper whose central claim is that capturing the full eigenstructure is necessary, the absence of an experiment that isolates the contribution of the eigenvectors (the "decorrelation" part) versus the eigenvalues (the "importance weighting" part) is a notable omission.

EoRA Has Not Been Compared Against a Matched-Budget Gradient-Based Baseline

The assumption or constraint. The paper positions EoRA against two classes of methods: fine-tuning-free methods (ZeroQuant-V2, Act-S, LQER, etc.) which are fast but less accurate, and gradient-based methods (ApiQ, LoRA, RILQ) which are more accurate but slow. The comparison against ApiQ uses ApiQ's full optimization procedure (layer-wise variant, "over hours"; Section 4.1), and the takeaway is that EoRA achieves comparable or better accuracy in a fraction of the time. However, the paper never asks the symmetrical question: what accuracy would gradient-based LoRA fine-tuning achieve if given the same compute budget (time or FLOPs) as EoRA's closed-form optimization? Specifically, what happens if you run a small number of gradient steps of LoRA on the same 64 calibration samples that EoRA uses, with the total optimization time capped at ~15 minutes?

The consequence. The comparison between EoRA and gradient-based methods is asymmetric. EoRA gets to use its full closed-form procedure (all calibration data, full rank), while ApiQ and LoRA are evaluated at their converged (or near-converged) states after much longer optimization. This makes EoRA look strictly dominant on the speed–accuracy Pareto frontier, but a matched-budget comparison might reveal regimes where a few gradient steps on LoRA outperform EoRA at the same wall-clock time, especially for tasks where the calibration data is informative and gradient signals are strong. The absence of this comparison weakens the claim that EoRA is categorically preferable to gradient-based methods for rapid task adaptation.

What evidence exists in the paper. The paper's LoRA fine-tuning experiments (Section 4.4, Tables 3, 11) use full fine-tuning: 3 epochs with batch size 64 on the target task's training set (e.g., the full ARC training set for ARC-C evaluation). These experiments show that EoRA initialization helps LoRA, but they do not answer the matched-budget question because the fine-tuning cost far exceeds EoRA's optimization time. The calibration time for EoRA is reported in Table 8 (e.g., 7.04 minutes for 32 samples, 8.03 minutes for 64 samples on LLaMA3-8B W4 MathQA). The paper never reports the accuracy of LoRA fine-tuned for 7–8 minutes on the same 32–64 calibration samples. The closest experiment is the data-ratio ablation in Table 12, which varies the fine-tuning dataset size (30%–100% of the training set) but keeps the number of training steps constant at 3 epochs — this varies data, not optimization budget.

Mitigation status. Not addressed. The paper does not acknowledge matched-budget comparison as a missing experiment. For a method whose primary claimed advantage over gradient-based approaches is speed ("in just a few minutes"), demonstrating that this speed actually translates to better accuracy at a given time budget (rather than just at convergence) is an essential validation that is absent.

The Scope of Empirical Validation Is Narrow: One Model Family, One Domain, No Production Deployment

The assumption or constraint. All accuracy experiments use the LLaMA2 and LLaMA3 model families (7B, 8B, 13B), with LLaMA3-70B used only for a single latency benchmark (Table 9). All tasks are English-language text benchmarks: language modeling (WikiText2), multiple-choice commonsense reasoning (ARC-Challenge), multiple-choice math reasoning (MathQA), open-ended math reasoning (GSM8K), summarization (CNN/DailyMail), and multi-task language understanding (MMLU). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4.1), but this claim is not tested on non-LLaMA architectures (e.g., Mistral, Falcon, Gemma), encoder-decoder models (T5), or non-text modalities (vision, speech, multimodal).

The consequence. It is unknown whether EoRA's eigenspace projection — which depends on the eigenstructure of transformer activation covariances — transfers to models with different architectural choices (different normalization schemes, activation functions, attention mechanisms) or different data modalities where activation covariance structure may differ qualitatively. If, for example, vision transformer activations exhibit different correlation patterns (due to spatial locality in image patches), the relative advantage of eigendecomposition over diagonal scaling may be different — potentially smaller if activation channels are less correlated, or larger if correlations are even stronger. Without multi-architecture validation, practitioners using non-LLaMA models have no evidence that EoRA will outperform simpler baselines.

What evidence exists in the paper. The paper tests LLaMA2-7B, LLaMA2-13B, LLaMA3-8B, LLaMA3.2-3B, and LLaMA3-70B (latency only) — all variations within the same architectural family (decoder-only transformer with SwiGLU FFN, rotary position embeddings, RMSNorm). There is no experiment on a non-LLaMA model. The pruning experiments test two methods (SparseGPT, Wanda) and the quantization experiments test one primary method (GPTQ) with one group-size variant (Table 16), but the compression techniques are not the primary dimension of generalization concern — the model architecture and task domain are.

Mitigation status. Partially acknowledged but not addressed. The paper's conclusion (Section 6) states that there are "potential extensions to new architectures and modalities" but treats this as future work rather than a limitation of the current validation. For a paper that presents EoRA as a general-purpose method for compressed LLM compensation, the restriction to one model family is a significant gap in the evidence.

EoRA Cannot Recover Accuracy When the Compressed Model Has Lost Fundamental Capability

The assumption or constraint. EoRA operates by approximating the compression error $\Delta W = W - \hat{W}$ in each layer, implicitly assuming that the information lost to compression is recoverable through a low-rank residual addition. When compression is so aggressive that the compressed model's behavior on a task collapses entirely — producing near-zero accuracy — EoRA's error approximation framework may lack a meaningful signal to approximate. The paper's results show this boundary: for LLaMA3-8B with combined 2:4 sparsity and 4-bit quantization (Table 6), GSM8K accuracy drops to 0.00% (the model cannot solve any problems), and EoRA recovers to only 10.16%. For 3-bit quantized LLaMA3-8B (Table 2), GSM8K drops to 0.45%, recovering to 11.90%. These recoveries are large in relative terms but leave the model far below usable accuracy for math reasoning.

The consequence. There exists a compression threshold beyond which EoRA cannot restore task competence, regardless of rank or calibration data quality. This threshold is task-dependent: math reasoning (GSM8K) collapses earlier than commonsense reasoning (ARC-C) under the same compression, likely because math reasoning requires more precise multi-step logical chains that are more sensitive to accumulated per-layer errors. For practitioners deploying aggressively compressed models, EoRA provides no guarantee of usable accuracy — and the paper provides no method for predicting, before applying EoRA, whether a given compression level is within the recoverable regime for a given task.

What evidence exists in the paper. The GSM8K results across compression levels demonstrate the threshold clearly. For 2:4 sparsity alone (Table 1): uncompensated accuracy is 2.12% (already near collapse), EoRA recovers to 13.95%. For 3-bit quantization alone (Table 2): 0.45% → 11.90%. For combined 2:4 + 4-bit (Table 6): 0.00% → 10.16%. These numbers show that EoRA can substantially multiply near-zero accuracy (e.g., ~25× for 3-bit GSM8K), but the absolute recovered accuracy (10–14%) remains far below the uncompressed baseline (36.23%) and may be insufficient for practical use. The paper does not discuss this capability boundary or provide guidance on how to predict it. The difficulty estimation framework from the related compute-optimal scaling literature (Section 3.3 of the prior analysis) is not applied or adapted here.

Mitigation status. The paper acknowledges implicitly that harder compression cases show larger relative gains from EoRA (e.g., "EoRA tends to offer greater accuracy recovery when addressing more aggressive compression settings," Section A.3) but frames this as a strength rather than acknowledging the absolute accuracy floor. The paper's problem formulation — customized compensation — assumes the compressed model is a fixed input, and EoRA's job is to improve it as much as possible. From this perspective, recovering 10 percentage points from a 0% baseline is a success of the method. But from a deployment perspective, a model that achieves 10% on GSM8K after compensation is still not useful for math reasoning, and a user needs to know this before investing time in applying EoRA. The paper does not provide tools for making this assessment.

7. Implications and Future Directions

How This Work Changes the Landscape

EoRA shifts the conversation around post-training LLM compression from a one-shot, monolithic decision to a decoupled, plug-and-play ecosystem where compression and task-specific accuracy are managed by separate, independently optimizable components. This is not a paradigm shift in the sense of overturning the foundations of model compression — the underlying compression algorithms (GPTQ, SparseGPT) and low-rank adapter architectures (LoRA) are unchanged. Rather, it is a reframing of the deployment workflow that has substantial practical consequences: instead of asking "which compression format should I use for all tasks?", practitioners can now ask "how much compression can I get away with on the backbone, knowing I can recover task-specific accuracy on demand with negligible overhead?"

The methodological contribution that enables this reframing — that projecting compression error into the activation eigenspace before SVD yields a closed-form solution to the layer-wise compression loss minimization — has three downstream effects on how the field thinks about error compensation:

1. It demonstrates that task-specific error compensation does not require gradient-based optimization. Prior to EoRA, the natural assumption (embodied by ApiQ, LoRA, and iterative methods like CALDERA and LRC) was that minimizing the activation-weighted compression loss with respect to low-rank matrices necessarily requires gradient descent or alternating optimization — that the objective is non-convex in the product BA and therefore iterative search is unavoidable. EoRA's Theorem 1 shows that a change of variables (projecting by Q√Λ) makes the problem convex in the transformed matrices, with a global optimum available via a single SVD. This is conceptually analogous to how the linear least-squares problem is solved in closed form by the normal equations — a reparameterization reveals hidden convexity. The practical consequence is that gradient-free methods can now occupy the accuracy tier previously reserved for gradient-based methods, collapsing the speed–accuracy tradeoff that defined the pre-EoRA landscape: ApiQ's 2.5-hour optimization achieves 34.21% ARC-C; EoRA's 15-minute closed form achieves 34.64% (Table 1). The speed advantage is not an incremental 2–3× but roughly an order of magnitude, which changes the calculus for when fine-tuning-free methods are appropriate.

2. It elevates the activation covariance from a diagnostic tool to a first-class optimization primitive. Before EoRA, activation statistics were used in compression either as post-hoc scaling heuristics (ASVD, LQER, SLiM — "high-magnitude channels are important, scale their errors up") or as calibration data for greedy layer-wise weight updates (GPTQ, SparseGPT). EoRA's innovation is to treat the full second-order structure of the activation covariance — its eigenvalues and eigenvectors — as the weighting function that makes unweighted SVD equivalent to activation-weighted approximation. This matters because it provides a principled answer to "which weight errors matter?" that is more nuanced than per-channel magnitude heuristics. The evidence that this nuance matters comes from Table 18: across seven activation-scaling methods that all use diagonal-only importance scores, EoRA's full-covariance approach leads by 1.5–3.5 points on 3-bit MathQA. The off-diagonal covariance structure — correlations between activation channels induced by attention mixing and FFN transformations — carries task-specific information that diagonal methods discard. This finding should redirect research on activation-aware compression away from designing better per-channel scaling formulas and toward methods that capture or approximate the full covariance structure.

3. It establishes that backbone preservation enables a deployment architecture (shared backbone + swappable adapters) whose overhead is quantifiably negligible, making the architecture practically viable for the first time. The concept of serving a single model with multiple LoRA adapters existed before EoRA (vLLM supports it), but the combination of EoRA's speed (adapters computed in minutes, not hours) and memory efficiency (4-bit quantized EoRA adds only 2–7% model size for substantial accuracy gains; Table 13, Figure 3b) makes the architecture economically compelling at a scale where it was previously aspirational. If adapters cost 20–30% extra memory and required hours of GPU time each, the shared-backbone paradigm would be a niche optimization for specialized serving infrastructure. EoRA's demonstration that rank-128 adapters quantized to 4-bit add 1.8% memory overhead on 2:4 pruned LLaMA3-8B while recovering 4.36 ARC-C points (30.11% → 34.47%; Table 13) means the marginal cost of supporting an additional task is small enough that the architecture can be adopted broadly, not just in resource-constrained edge deployments.

This work resolves a tension in the low-rank compensation literature between methods that are fast but task-agnostic (ZeroQuant-V2, naive SVD) and methods that are task-specific but slow (ApiQ, LoRA fine-tuning). The tension was: task-specificity requires calibration data, and using calibration data effectively was assumed to require gradient-based optimization. EoRA shows that the assumption is false — a single pass through the calibration data to compute the activation covariance, followed by eigendecomposition and SVD, extracts sufficient task-specific information to match or exceed gradient-based methods on most tasks. This explains why ApiQ, despite directly optimizing the correct objective (Equation 6) via gradient descent, does not consistently outperform EoRA: the optimization landscape for the layer-wise compression loss appears to be sufficiently well-behaved that the closed-form eigenspace solution lands near the global optimum, and gradient-based refinement provides only marginal additional benefit (and sometimes none, as on ARC-C where EoRA beats ApiQ by 0.43 points at rank 128). This suggests that future work on low-rank compensation should invest in better activation covariance estimation (more informative priors, better calibration data selection, multi-task covariance sharing) rather than better optimizers — a shift in research priorities that the paper's results directly motivate.

Follow-Up Research This Work Enables

Eigenvalue-only ablation to isolate the contribution of decorrelation vs. importance weighting. The paper's central theoretical claim is that the full eigenspace projection Q√Λ — encompassing both rotation into the decorrelated eigenbasis (Q) and scaling by eigenvalue importance (√Λ) — makes SVD minimize the true compression loss. However, the paper never measures whether the rotation (Q) is actually necessary, or whether scaling the raw error ΔW by √Λ alone (a diagonal operation that uses the eigenvalues but ignores the eigenvectors, and thus preserves the original input basis) would achieve most of the gain. A targeted experiment: on 3-bit quantized LLaMA3-8B MathQA (where the EoRA–Act-S gap is 3.69 points; Table 2), compare EoRA against a method that (a) scales columns of ΔW by √Λii without rotating, and (b) rotates via Q without scaling (i.e., uses uniform eigenvalue weighting). If the scaling-alone variant recovers >80% of EoRA's gain over Act-S, the practical takeaway is that eigenvalue estimation (a cheaper diagonal operation) is the critical ingredient, and full eigendecomposition may be overkill for most use cases. If rotation is essential, future work should focus on efficient approximations to the eigenbasis (e.g., power iteration to extract top-r eigenvectors only, avoiding full O(k³) decomposition).

Eigendecomposition cost scaling and mitigation for large models. The paper reports ~15-minute optimization time for LLaMA3-8B but provides no timing breakdown or scaling analysis. A direct follow-up: measure wall-clock time for eigendecomposition, SVD, and matrix multiplication separately on LLaMA3-8B, then project to 70B and 405B scales using the known cubic scaling of eigendecomposition. For 70B, the intermediate FFN dimension is 28,672, so eigendecomposition cost per FFN layer is roughly (28,672/14,336)³ = 8× higher than 8B, and there are more layers. Total eigendecomposition time could exceed several hours — competitive with or slower than gradient-based ApiQ. If this projection is confirmed, the follow-up becomes: can randomized eigendecomposition (e.g., using only top-k eigenvectors via randomized SVD, with k ≪ full dimension) preserve EoRA's accuracy while reducing the eigendecomposition cost from O(k³) to O(k² * r_eff*) where r_eff is the number of retained eigencomponents? The paper's rank scaling results (Figure 2) show that rank-64 EoRA already captures most of the available gain on some tasks (e.g., ARC-C: 33.10% at rank 64 vs. 34.64% at rank 128), suggesting that the top few dozen eigen-directions dominate — a randomized eigendecomposition keeping only the top 64–128 eigencomponents could dramatically reduce cost while maintaining accuracy.

Multi-adapter deployment benchmark with dynamic adapter swapping. The paper's problem formulation centers on the shared-backbone + task-specific-adapter architecture (Figure 1) but provides zero measurements of this architecture in practice. A deployment-focused follow-up: serve a single 3-bit quantized LLaMA3-8B backbone with EoRA adapters (rank 128, 4-bit quantized) for three tasks (ARC-C, MathQA, GSM8K) on an inference server, and measure (a) storage savings vs. maintaining three separate task-specific compressed models, (b) throughput (tokens/second) under mixed-task request streams with adapter swapping, (c) adapter loading latency when switching tasks, and (d) whether the 1.4× kernel speedup (Table 9) persists under concurrent multi-adapter serving. The key comparison is against the alternative deployment strategy: three separately fine-tuned compressed models (e.g., using LoRA + EoRA initialization as in Table 11) loaded simultaneously. If the shared-backbone approach reduces GPU memory by 40–60% (because only one copy of the backbone weights is stored vs. three) while maintaining >90% of the per-task throughput, the practical case for backbone preservation is validated. If adapter swapping introduces latency spikes that make interactive use impractical, the architecture may be limited to batch processing scenarios.

Cross-architecture and cross-modal generalization of the eigenspace projection approach. All experiments use LLaMA-family decoder-only transformers. An important stress test: apply EoRA to a structurally different architecture — e.g., an encoder-decoder model (T5, Flan-T5) where activation covariance patterns may differ due to bidirectional encoder attention, or a vision transformer (ViT) where activation channels correspond to spatial patch embeddings with strong local correlations. On T5-XXL compressed to 3-bit, compare EoRA against Act-S on summarization (CNN/DailyMail, XSum) and question answering (SQuAD). The hypothesis: EoRA's advantage over diagonal scaling methods should be larger in architectures or modalities where activation channels are more strongly correlated (e.g., vision transformers with overlapping patch embeddings), and smaller where channels are more independent (e.g., after extensive LayerNorm that whitens activations). If EoRA's advantage disappears on T5, the method's generalizability claim is undermined; if it persists or grows on ViT, the method's scope expands to multimodal models. A negative result on some architectures would be equally informative, helping to characterize the conditions under which full eigendecomposition is worth its computational cost.

Matched-budget comparison against few-step gradient-based LoRA. The paper compares EoRA (15 minutes, closed-form) against ApiQ (2.5 hours, gradient-based) and LoRA (3 epochs, gradient-based) but never against LoRA run for the same 15-minute wall-clock budget. A targeted experiment: on 3-bit quantized LLaMA3-8B, run LoRA fine-tuning (rank 128, learning rate 1e-5) on the same 64 calibration samples EoRA uses, but stop training at the 15-minute mark. Compare accuracy against EoRA on ARC-C, MathQA, and GSM8K. If LoRA at 15 minutes underperforms EoRA substantially, the closed-form advantage is robust. If LoRA at 15 minutes matches or exceeds EoRA, the practical advantage shifts: EoRA is still preferable for its determinism (no random seed sensitivity, no hyperparameter tuning) but not for its speed-accuracy Pareto frontier. Additionally, vary the calibration size (32, 64, 128 samples) and training time (5, 15, 30 minutes) to map out the full speed–data tradeoff surface for both methods. The paper's calibration-size ablation (Table 8) provides EoRA's side of this surface; adding LoRA's side would complete the picture.

Difficulty-aware rank allocation across layers and tasks. EoRA applies a uniform rank r to all linear layers, but the compression error and activation covariance structure likely vary across layers (early vs. late, attention vs. FFN). A natural extension: use the eigenvalues from the eigendecomposition to guide heterogeneous rank allocation — layers where the top few eigenvalues dominate the trace of Λ (indicating low effective rank in the activation covariance) need lower compensation rank than layers where eigenvalues decay slowly (indicating high-dimensional activation structure). The paper already has the necessary per-layer eigenvalue spectra from step 2 of Algorithm 1; an allocation heuristic could set each layer's rank proportional to the number of eigenvalues needed to capture 95% of the trace of Λ. Compare uniform-rank EoRA against eigenvalue-proportional EoRA at the same total parameter budget: if heterogeneous allocation improves accuracy at equal parameter count, the eigenvalue spectrum becomes a practical tool for optimizing the accuracy–overhead tradeoff beyond what uniform rank provides. A negative result — eigenvalue-proportional allocation does not help — would suggest that compression error magnitude, not activation structure, is the binding constraint on rank requirements, which would simplify deployment by justifying uniform rank as near-optimal.

Practical Applications and Downstream Use Cases

On-demand task adaptation for multi-tenant model serving. A cloud inference provider serves a single compressed LLaMA3-8B backbone (e.g., 3-bit quantized, 4.63 GB) to hundreds of customers, each with different task requirements — one needs strong math reasoning, another needs commonsense QA, a third needs summarization. Without EoRA, the provider either serves the uncompensated 3-bit model at degraded accuracy on all tasks, or maintains separate fine-tuned model copies per customer (4.63 GB × N, memory-prohibitive). With EoRA: the provider pre-computes task-specific adapters offline (rank 128, 4-bit quantized, ~0.15 GB each; Table 13) for the most common task categories. At serving time, each request specifies its task, the appropriate adapter is loaded alongside the shared backbone (supported by vLLM's multi-adapter inference), and inference proceeds with the EoRA kernel (1.4× speedup over FP16 at 3-bit; Table 9). The marginal storage cost per additional task is 3% of the backbone size, and adapter computation is a one-time cost of ~15 minutes per task. The accuracy benefit: on MathQA, the 3-bit backbone with EoRA adapter achieves 29.11% vs. 22.37% uncompensated (Table 2) — a 30% relative improvement that directly translates to user-perceived quality. This architecture is deployable today with existing infrastructure (vLLM + custom CUDA kernel) and the paper's provided code.

Rapid prototyping of compressed models for new downstream tasks. A research team building a specialized QA system for a new domain (e.g., legal document analysis, biomedical literature) wants to deploy a compressed LLM but does not have the budget or time for full fine-tuning. They take an off-the-shelf 4-bit quantized LLaMA3-8B, collect 64 domain-specific calibration examples (roughly 131K tokens), and run EoRA — within 8 minutes (Table 8, 64-sample calibration time), they have a domain-adapted compressed model. The accuracy recovery is immediate: on MMLU, which spans diverse knowledge domains, EoRA recovers 4-bit LLaMA3.2-3B from 24.16% to 52.93% (Table 15), nearly matching the full-precision baseline of 54.19%. If the recovered accuracy is insufficient, the EoRA matrices serve as initialization for full LoRA fine-tuning (Section 4.4): on 4-bit quantized LLaMA3-8B MathQA, EoRA initialization + fine-tuning achieves 56.04% vs. QLoRA's 51.42% (Table 3), a 4.62-point improvement that would require substantially more training time to achieve from random initialization. The workflow — 8-minute compensation → evaluate → optionally fine-tune — is fast enough to support iterative development cycles where multiple compression configurations and calibration sets are tested in a single day.

Edge deployment with dynamic accuracy–latency tradeoff. A mobile application uses an on-device compressed LLM (e.g., 2:4 pruned LLaMA3-8B, 9.12 GB) for interactive tasks. Network conditions and battery status vary throughout the day. With EoRA, the application can ship with the compressed backbone pre-installed and a set of task-specific adapters (e.g., math reasoning, creative writing, factual QA) stored in the app bundle. When the user asks a math question, the math adapter is loaded (rank 128, 4-bit quantized: +0.16 GB, +1.8% memory; Table 13), boosting GSM8K accuracy from 2.12% to 13.95% (Table 1). When battery is low or the user is on a metered connection, the adapters are disabled and the raw compressed model is used — the accuracy tradeoff is controlled entirely on-device without any server round-trips or model reloading. The rank can also be varied dynamically: rank 64 provides 10.77% GSM8K with lower latency overhead (Figure 2), while rank 512 provides 23.28% GSM8K at higher cost. The paper's finding that EoRA adapters are robust to quantization (Table 13) means the app developer can further compress the adapters to 3-bit for a smaller download size, accepting a small accuracy penalty (rank 512: 41.89% → 40.35% ARC-C, a 1.54-point drop vs. 30.11% uncompensated baseline). This deployment model is not hypothetical — the paper provides pre-computed adapters and kernel code, and the inference speed measurements (Table 9) confirm that the latency overhead is manageable with the custom kernel.

Cost-efficient data generation for self-improvement pipelines. A team building a self-improvement loop (e.g., STaR, ReST^EM style) needs to generate high-quality reasoning traces from a compressed model to use as training data for the next iteration. Running full-precision generation is expensive; running the raw compressed model produces mostly incorrect traces (e.g., 3-bit LLaMA3-8B achieves 0.45% GSM8K, generating 99.55% incorrect solutions; Table 2). The standard solution — fine-tune the compressed model on existing correct traces — is circular. EoRA offers a lightweight alternative: compute task-specific adapters from a small set of calibration examples (which need not be correct — they just need to be in-distribution), apply them during generation to boost accuracy (3-bit GSM8K: 0.45% → 11.90%; Table 2), and use the improved traces (now ~8× more likely to be correct) as training data. The EoRA adapters are then discarded or updated for the next iteration. The cost: 8 minutes of EoRA computation per iteration plus the inference cost of the compensated model (with the 1.4× speedup kernel; Table 9). The benefit: an 8× increase in correct trace yield for the same inference budget, which could substantially accelerate self-improvement convergence. The paper does not explore this application, but the components — fast adapter computation, backbone preservation, quantized adapter storage — are all directly applicable.