SliceGPT: Compress Large Language Models by Deleting Rows and Columns

ArXiv: 2401.15024

🎯 Pitch

SliceGPT deletes entire rows and columns from transformer weight matrices instead of zeroing individual weights, achieving structured sparsity that directly reduces the embedding dimension and immediately shrinks both memory and compute on existing hardware without custom kernels. It removes up to 25% of LLM parameters while keeping 99% of zero-shot accuracy on 70B models, based on a new insight that orthogonal transforms can reorient the network’s internal representations so PCA can safely discard minor components.

---

1. Executive Summary

This paper introduces SliceGPT, a new post-training sparsification scheme for large language models that replaces each weight matrix with a smaller dense matrix by reducing the network's embedding dimension — effectively deleting entire rows and columns rather than zeroing individual weights. The method is built on a new insight the authors call computational invariance, which shows that orthogonal transformations can be applied to each weight matrix in a transformer without changing the model's output, enabling the signal between blocks to be projected onto its principal components via PCA so that minor components can be sliced away with minimal degradation. Evaluated on OPT, LLAMA-2, and Phi-2 models using the WikiText-2 generation benchmark and five zero-shot tasks, SliceGPT can remove up to 25% of model parameters while maintaining 99% of zero-shot accuracy for LLAMA-2 70B and OPT 66B — and on consumer GPUs reduces total compute for LLAMA-2 70B inference to 64% of the dense model's cost, establishing that structured dimensionality reduction achieves both speedup and competitive accuracy compared to 2:4 sparsity patterns only when the underlying model's embedding spectrum exhibits sufficient concentration of variance in the dominant principal components.

2. Context and Motivation

The Core Problem: Model Deployment Is the Bottleneck, Not Training

The paper addresses a fundamental tension in the current large language model ecosystem. Organizations routinely train models with tens or hundreds of billions of parameters — OPT 66B, LLAMA-2 70B, and larger — but actually using these models for inference is where the real resource pain lives. The paper opens by noting a shift toward the foundation model paradigm: pre-train once, reuse for many tasks. But this paradigm carries an implicit asymmetry — training is a one-time capital expense, while inference is an ongoing operational expense that scales with every user query.

The operational costs come in two forms that the paper emphasizes (Section 1):

• Memory: Many models require multiple GPUs simply to hold the parameters in device memory. The paper's Quadro RTX6000 experiments show LLAMA-2 70B needs 7 GPUs for the dense model, while OPT 66B needs 6 — both exceeding what any single 24GB consumer GPU can handle. This is not just a hardware cost issue; it's an accessibility issue. Researchers and developers without datacenter-scale resources are locked out of the largest models entirely.
• Compute: Because LLMs are autoregressive, generating a single response requires running the model's forward pass once per output token. A 70B-parameter model that takes 125ms per token (Table 2) needs over 16 seconds to generate a modest 128-token response — and this is on 4 A100 GPUs. The paper quantifies this in GPUms: the product of GPU count and time per token. LLAMA-2 70B on consumer GPUs costs 1764 GPUms per token. Over millions of inference requests, this number dominates the total cost of ownership.

These deployment costs motivate what the paper calls "post-training techniques referred to as model compression" (Section 1). The goal is to take an already-trained model and make it cheaper to run without retraining it from scratch — which would defeat the purpose of the foundation model paradigm.

Why Existing Compression Approaches Don't Fully Solve the Problem

The paper situates itself within a mature landscape of model compression techniques, surveying four categories: distillation, tensor decomposition / low-rank factorization, pruning, and quantization (Section 1 and Section 2.2). But each category has structural limitations that SliceGPT is designed to overcome, and understanding these limitations is crucial to appreciating why the paper's approach is not just another pruning method.

Unstructured Pruning: Speedup That Remains Theoretical

The simplest approach is magnitude-based pruning: take the trained weight matrices, zero out the weights with the smallest absolute values, and optionally fine-tune the remaining weights to compensate (Han et al., 2016; Zhu & Gupta, 2017). This is computationally cheap to apply — you just need a sort-and-threshold operation — but the paper identifies a fundamental mismatch with modern hardware (Section 2.2). When you zero out individual weights at random positions in a matrix, the resulting sparse pattern is unstructured — the non-zero elements are scattered irregularly. Modern GPUs are designed around dense linear algebra operations that access memory in contiguous blocks. Sparse matrix multiplication with irregular patterns requires special handling that often eliminates any theoretical FLOP reduction advantage. As the paper notes: "achieving end-to-end speed improvements through unstructured pruning is a demanding task" (Section 2.2).

The paper also examines more sophisticated pruning approaches derived from the Optimal Brain Surgeon framework (Hassibi et al., 1993; LeCun et al., 1989). Instead of simply removing the smallest weights, OBS calculates which weights have the least impact on the loss function using second-order information — the Hessian matrix of the loss with respect to the weights — and compensates for the removal by adjusting the remaining weights. This is mathematically principled but computationally pathological: "OBS is impractical for models with a few million parameters due to the need to calculate and store the inverse of the Hessian matrix" (Section 2.2). LLMs have weight matrices with more than $10^8$ parameters per layer. The $D \times D$ Hessian for a single layer would require storing a matrix with over $10^{16}$ entries — utterly infeasible.

Some variants have attacked this via approximation: WoodFisher (Singh & Alistarh, 2020) approximates the inverse Hessian, and Optimal Brain Compression (Frantar & Alistarh, 2022) applies OBS layer-wise, treating each layer independently. But even these fall short for LLM-scale models. The breakthrough for LLM-scale pruning came with SparseGPT (Frantar & Alistarh, 2023), which adapted the column-by-column updating scheme from the GPTQ quantization method to the pruning setting. SparseGPT can sparsify LLMs using both unstructured and semi-structured patterns, but critically, the paper treats SparseGPT's 2:4 semi-structured variant as its primary baseline — and this choice is itself revealing.

Semi-Structured Sparsity (2:4): A Hardware-Aware Compromise

The 2:4 sparsity pattern (Mishra et al., 2021) is a specific structured pruning arrangement: in every contiguous block of 4 weight entries, exactly 2 must be zero. This pattern is directly supported by NVIDIA's sparse tensor core instructions, meaning it actually translates into runtime speed improvements — unlike unstructured sparsity. But Figure 1 (middle panel) illustrates the tradeoff: the weight matrix $W$ has a regular pattern of zeros, but the signal matrix $X$ is still dense and full-sized. The sparsity is in the weights only; the activations flowing through the network remain at the original embedding dimension.

The paper's empirical results (Table 1) show that SparseGPT 2:4 does achieve reasonable perplexity on LLAMA-2 models — for LLAMA-2 70B, SparseGPT 2:4 achieves 4.98 perplexity compared to 3.32 for the dense model. But there are two problems. First, the degradation is non-trivial: a 1.66-point perplexity increase for a 70B model. Second, and more subtly, 2:4 sparsity does not reduce the number of GPUs needed to fit the model in memory, nor does it reduce the size of the activations between layers. You still need to store and move the same amount of data for $X$; you just compute the weight matrix multiplication faster. The paper's throughput experiments in Appendix A.6 and A.7 confirm this nuance: SparseGPT 2:4 achieves a $1.32\times$ speedup on LLAMA-2 70B dense matrix multiplications (Table 12), but the model still requires 2 GPUs — the same as the dense model in the throughput benchmark.

Low-Rank Factorization: Theoretical Elegance, Practical Limitations

The paper briefly addresses low-rank approximation (Section 2.2), where each weight matrix $W$ of shape $D \times D$ is replaced with the product of two smaller matrices: $A$ of shape $D \times r$ and $B$ of shape $r \times D$, where $r < D/2$ is required for actual compression. This approach decomposes the linear transformation into a bottleneck, reducing both parameters and FLOPs. The idea is well-established in the literature (Hu et al., 2021; Mahabadi et al., 2021; Noach & Goldberg, 2020).

But the paper identifies a critical practical constraint: "To achieve compression, the inner dimension must be smaller than half of the original dimension." This is a high bar. If the original matrix is $D \times D$, replacing it with two matrices of shapes $D \times r$ and $r \times D$ yields a total parameter count of $2Dr$. For compression, we need $2Dr < D^2$, or $r < D/2$. But the rank $r$ must be chosen before decomposing the matrix, and there's no guarantee that a pre-trained weight matrix is well-approximated by such a low-rank factorization without substantial fine-tuning. The paper notes that low-rank methods are "usually followed by a fine-tuning step" — and for LLMs, fine-tuning at scale is itself expensive. SliceGPT's selling point is that it avoids this: it replaces each weight matrix with a single smaller one without needing recovery fine-tuning to maintain performance (though recovery fine-tuning is optionally applied and shown to help).

Structural Pruning in CNNs: An Analogous but Non-Transferable Literature

The paper draws an explicit parallel to filter/channel pruning in convolutional neural networks (Section 2.2), citing methods like ThiNet (Luo et al., 2017; He et al., 2017) that apply linear transformations between layers and delete entire channels. These methods are philosophically similar to SliceGPT — they reduce the network's width rather than introducing sparse patterns — but they operate on models that are "several orders of magnitude smaller." The VGG16 network, at 138M parameters, is comparable to only the very smallest OPT model (125M). The paper argues that methods requiring "extensive fine-tuning" or "outer-loops to select regularization parameters" are impractical for LLMs. The sheer scale changes the design constraints: SliceGPT must work as a one-shot method on models 100–500× larger, running on a single GPU in a few hours.

LLM-Specific Structural Pruning: The Recovery Fine-Tuning Dependency

The paper acknowledges recent work on structured pruning specifically for LLMs: LLM-Pruner (Ma et al., 2023a) removes connected structures from an LLM, and the contemporaneous LLM Surgeon (van der Ouderaa et al., 2023) interweaves pruning with recovery fine-tuning. These methods share SliceGPT's goal of removing structured components (not individual weights), but the paper positions SliceGPT as distinct in its one-shot capability. The core results in Table 1 and Figure 5 are presented without recovery fine-tuning, and the paper explicitly states: "we compress large models using a single GPU in just a few hours and maintain competitive performance on generation and downstream tasks even without RFT" (Section 1). Recovery fine-tuning is then explored as an optional post-processing step that can recover additional accuracy (Figure 6), but it is not required for SliceGPT to achieve its headline results.

The Unstated Gap: Nobody Was Reducing Embedding Dimension Directly

Reading across these prior approaches, a pattern emerges: all existing methods — pruning, low-rank factorization, structural pruning with fine-tuning — either leave the embedding dimension $D$ unchanged or require additional data structures that create overhead. Unstructured sparsity reduces the effective number of weights but keeps matrices the same size and adds irregular memory access patterns. 2:4 sparsity reduces computation but not activation memory. Low-rank factorization changes the matrix structure but requires two matrices where one existed before. Structural pruning with RFT reduces dimensions but needs expensive retraining loops.

SliceGPT's distinctive contribution is to ask: what if we could simply make the weight matrices physically smaller, and simultaneously shrink the activations flowing between them? This is the vision illustrated in Figure 1 (right): after applying an orthogonal transformation $Q$, the sparsity is concentrated entirely in the bottom rows of $Q^\top W$, and the corresponding columns of $XQ$ can be removed entirely. The result is not a sparse matrix but a smaller dense matrix — one that uses standard dense matrix multiplication kernels with no sparsity overhead, and simultaneously reduces the embedding dimension so that future layers process smaller signals. This is a structural compression that operates at the network architecture level rather than the weight value level.

Why This Gap Matters

The absence of a dimensionality-reduction approach for LLMs is not an accident — it reflects a genuine technical obstacle. If you randomly delete columns from a weight matrix in a trained network, you destroy the carefully learned representations and the network's output becomes garbage. The paper confirms this in Section 4: "We initially planned to compare our results against a scheme that pruned columns (or rows) with the smallest norm but found that this baseline was very poor, with the WikiText-2 perplexity of the model soaring into the 1000s after pruning just a few columns."

The technical obstacle is that layers in a transformer are tightly coupled. The output of block $\ell$ becomes the input to block $\ell + 1$ via residual connections and normalization. If you remove columns from the output of block $\ell$, the input to block $\ell + 1$ now has the wrong dimensionality. If you remove rows from the input of block $\ell + 1$, the matrix multiplication is ill-defined. The network's architecture creates a rigid constraint: every signal passing between blocks must have the same dimension $D$, and that dimension must match the weight matrices on both sides.

SliceGPT's technical contribution — the computational invariance that is the paper's centerpiece — is the insight that breaks this constraint. By showing that orthogonal transformations can be absorbed into the weight matrices on both sides of each RMSNorm operation without changing the network's output, the paper creates a degree of freedom that didn't exist before: the network can be rotated into a basis where some dimensions carry negligible information, and those dimensions can then be deleted without breaking the architectural coupling between layers.

How the Paper Positions Itself

The paper makes three explicit positioning claims that frame its contribution relative to prior work:

1. It is a one-shot method. The abstract and Section 1 emphasize that SliceGPT works without recovery fine-tuning — compressing LLAMA-2 70B in ~3.5 hours on a single H100 GPU (Table 3). This contrasts with methods like LLM-Pruner and LLM Surgeon that require continued training or interleaved pruning-fine-tuning loops.

2. It produces smaller dense matrices, not sparse ones. The paper explicitly contrasts its approach with both unstructured and 2:4 sparsity in Figure 1, and the implication is practical: dense matrix multiplication runs efficiently on all hardware with no special kernel support. The inference speedup (Table 2) comes from running standard PyTorch operations on smaller matrices, not from exploiting sparsity-aware hardware instructions.

3. It reduces the embedding dimension, not just the weight count. This is perhaps the most distinctive positioning. The paper emphasizes that SliceGPT "replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network" (abstract). Because both the weights and the activations shrink, the compression benefits compound: a 25% reduction in $D$ means each subsequent layer processes 25% fewer activations, reduces its weight matrices by approximately $1 - (0.75)^2 = 43.75\%$ (since weight matrices are roughly $D \times D$), and requires less memory for intermediate results. This compounding is not available to methods that only sparsify weights while leaving activations at full size.

The paper also implicitly positions SliceGPT as complementary to other compression methods, noting in Section 5 that "complementary methods including quantization and structural pruning could be used" on top of sliced models. This suggests SliceGPT is not competing to replace the entire compression pipeline but rather adding a new axis — embedding dimension reduction — that can be combined with quantization (reducing bit precision), additional pruning (sparsifying the already-smaller matrices), and knowledge distillation.

The Theoretical Anchor: Why Orthogonal Transformations?

A subtle but important aspect of the motivation is why orthogonal transformations are the right tool for this job, even though the paper doesn't fully develop this argument in Section 2. When you apply a transformation $Q$ to the signal matrix $X$, you want the information content to be preserved so that the network's computation remains unchanged. Orthogonal transformations are norm-preserving: $\|Qx\| = \|x\|$, which means the scale of activations doesn't change. This matters because transformer networks are carefully calibrated — normalization layers, residual connections, and attention softmax operations all assume activation magnitudes within certain ranges. If the transformation changed the norm of the signals, the LayerNorm/RMSNorm operations would need to compensate, and the proof of invariance would break.

More fundamentally, PCA — which the paper uses to compute the transformation matrices $Q_\ell$ — is defined in terms of orthogonal transformations. PCA finds the orthogonal basis that diagonalizes the covariance matrix $X^\top X$, ordering the basis vectors by the amount of variance they capture. The eigenvalues (variances) tell you which dimensions matter most. If the eigenvalue spectrum decays rapidly — which the paper's spectrum analysis in Appendix A.4 explores — then a small number of principal components capture most of the signal energy, and the remaining components can be deleted with minimal reconstruction error. This is standard PCA theory, but applying it inside a transformer network, block by block, without changing the network's output requires the computational invariance that the paper proves in Theorem 1.

Summary of the Gap and the Paper's Response

The pre-existing compression landscape offered a clear tradeoff: you could either (a) use unstructured sparsity, which gives good accuracy but no speedup; (b) use semi-structured sparsity like 2:4, which gives some speedup but limited memory reduction and non-trivial perplexity degradation; or (c) use low-rank factorization with fine-tuning, which changes the weight matrix structure but requires expensive retraining and may not compress if the required rank is too high. No existing method systematically reduced the embedding dimension itself — the $D$ that propagates through every layer of the network — without breaking the model.

SliceGPT fills this gap by showing that orthogonal transformations reveal a compressed representation of the signal within each block, that these transformations can be absorbed into the surrounding weight matrices without changing the model's computation, and that the compressed representation enables physically deleting rows and columns to produce smaller, dense matrices that run faster on standard hardware with no special kernel support. The paper's claim to novelty rests on this combination: computational invariance (the theoretical insight) plus PCA-based dimensionality reduction (the practical mechanism) plus structured deletion of rows and columns (the physical compression).

3. Technical Approach

3.1 Reader Orientation

This is primarily a systems and methods paper whose core idea is that pre-trained transformer networks possess a previously unrecognized algebraic freedom — the ability to absorb orthogonal transformations into their weight matrices — which can be exploited to rotate the network into a basis where Principal Component Analysis reveals redundant dimensions that can be physically deleted, producing smaller dense matrices that run faster on standard hardware without specialized sparse kernels.

The problem SliceGPT solves is how to reduce the embedding dimension of an already-trained transformer without changing its output distribution, and the solution takes the shape of a three-stage pipeline: first convert all LayerNorm operations to RMSNorm to enable the invariance, then compute block-specific orthogonal transformations from calibration data using PCA, and finally delete the rows/columns corresponding to the smallest principal components across all affected weight matrices.

3.2 Big-Picture Architecture (Diagram in Words)

The SliceGPT compression system has five major components arranged in a sequential pipeline:

1. LayerNorm-to-RMSNorm conversion — a preprocessing step that absorbs the mean-subtraction and per-channel scaling of LayerNorm into the adjacent weight matrices, transforming every normalization operation in the network into a pure RMSNorm that divides each row by its L2 norm. This step is required because the computational invariance theorem holds only for RMSNorm-connected networks.

2. Calibration data forward pass — the converted network runs on a small set of calibration sequences (from WikiText-2 or Alpaca training sets), and at each RMSNorm output, the signal matrix $X_\ell$ is collected. These signals encode the distribution of activations that the network produces on representative data.

3. Per-block PCA computation ($Q_\ell$ construction) — for each block $\ell$, the covariance matrix $C_\ell = \sum_i X_{\ell,i}^\top X_{\ell,i}$ is computed from the collected signals, and its eigendecomposition yields an orthogonal matrix $Q_\ell$ whose columns are the principal components of the signal distribution, ordered by decreasing eigenvalue. This matrix defines the rotation that concentrates information into the leading dimensions.

4. Orthogonal transformation absorption — each $Q_\ell$ is absorbed into the surrounding weight matrices according to Theorem 1: input matrices are pre-multiplied by $Q_\ell^\top$, output matrices are post-multiplied by $Q_\ell$, and a small linear layer $Q_{\ell-1}^\top Q_\ell$ is inserted into the residual connection to maintain dimensional consistency between blocks with different rotations.

5. Slicing (dimension deletion) — a deletion matrix $D$ (containing $D_{\text{small}}$ columns of the $D \times D$ identity matrix) removes the rows/columns corresponding to the smallest eigenvalues from every affected weight matrix. The result is a network with all weight matrices physically smaller by the same amount, and all inter-block signals reduced in dimension from $D$ to $D_{\text{small}}$.

Information flows linearly through these components: a pre-trained model enters the pipeline → LayerNorm blocks are decomposed and absorbed → calibration data generates signal statistics → PCA produces rotation matrices → rotations are fused into weights → minor PCA dimensions are sliced away → a smaller, dense model is output.

3.3 Roadmap for the Deep Dive

• First, the LayerNorm-to-RMSNorm conversion (Section 3.2 in the paper), because the computational invariance theorem — the theoretical foundation for everything that follows — requires RMSNorm-connected networks, and applying the conversion first makes the entire subsequent derivation valid for standard transformer architectures.

• Second, the computational invariance theorem itself (Section 3.1 in the paper, Theorem 1), which establishes that orthogonal transformations can be absorbed into weight matrices without changing the network's output. This is the central insight that makes the whole approach possible.

• Third, the extension to per-block transformations (Section 3.3), which addresses the practical reality that signal distributions differ across layers and therefore need different rotations — and introduces the residual connection modification that this extension requires.

• Fourth, the PCA-based computation of the orthogonal matrices $Q_\ell$ (Section 3.3), including the covariance matrix construction, eigendecomposition, and the calibration dataset design choices.

• Fifth, the slicing mechanism (Section 3.4), which formalizes how the deletion matrix $D$ removes dimensions from the transformed network and which weight matrices are affected in which way.

Each component builds on its predecessor: conversion enables invariance, invariance enables rotation, per-block rotation enables PCA, PCA reveals which dimensions to delete, and slicing performs the deletion.

3.4 Detailed, Sentence-Based Technical Breakdown

LayerNorm-to-RMSNorm Conversion

The computational invariance that SliceGPT exploits (Theorem 1) applies only to networks connected by RMSNorm, not the more common LayerNorm. Since most pre-trained transformers — including OPT, LLAMA-2, and Phi-2 — use LayerNorm, the first step of the pipeline is a mathematically equivalent rewrite of the network that converts every LayerNorm into an RMSNorm by absorbing the LayerNorm's linear components into adjacent weight matrices.

A LayerNorm block (Equation 1) performs the following operations on an input matrix $X$ of shape $N \times D$, where $N$ is sequence length and $D$ is embedding dimension:

$\text{LayerNorm}(X) = \text{RMSNorm}(X M) \, \text{diag}(\alpha) \sqrt{D} + 1_N \beta^\top$

Here $M = I - \frac{1}{D} 11^\top$ is a $D \times D$ matrix that subtracts the mean from each row of $X$ (it projects out the all-ones direction), $\alpha$ is a learned $D$-dimensional scale vector, $\beta$ is a learned $D$-dimensional offset vector, and $1_N$ is an $N$-dimensional vector of ones. The RMSNorm operation divides each row by its Euclidean norm: $x \leftarrow x / \|x\|$. The factor $\sqrt{D}$ compensates for the scaling difference between RMSNorm and LayerNorm (since LayerNorm normalizes by standard deviation while RMSNorm normalizes by root-mean-square).

What the conversion does: The paper decomposes LayerNorm into two steps that can each be absorbed into adjacent linear layers. The mean-subtraction matrix $M$ is associative with matrix multiplication, so applying $M$ to $X$ before a weight matrix $W$ is equivalent to applying it to $W$ instead: $X (M W) = (X M) W$. Similarly, the diagonal scaling $\text{diag}(\alpha)$ can be absorbed: $(X \text{diag}(\alpha)) W = X (\text{diag}(\alpha) W)$. This means:

1. For the output side of each block: The output weight matrix $W_{\text{out}}$ (which produces the signal that will be normalized before the next block) is post-multiplied by $M$: $\tilde{W}_{\text{out}} = W_{\text{out}} M$. This handles the mean subtraction that LayerNorm would have performed on the block's output.

2. For the input side of the next block: The input weight matrix $W_{\text{in}}$ (which receives the normalized signal) is pre-multiplied by the scale factors from the previous LayerNorm: $\tilde{W}_{\text{in}} = \text{diag}(\alpha) W_{\text{in}}$. This handles the per-channel rescaling.

3. For the embedding matrix: The very first LayerNorm (after embeddings) requires $W_{\text{embd}}$ to have mean subtraction applied: $\tilde{W}_{\text{embd}} = W_{\text{embd}} M$, and the first block's input matrix gets the scale absorption.

4. For the LM head: The final LayerNorm's scale factors are absorbed into $W_{\text{head}}$: $\tilde{W}_{\text{head}} = \text{diag}(\alpha_{\text{last}}) W_{\text{head}}$.

5. Bias terms: The offset $\beta$ from each LayerNorm is added to the bias of the subsequent linear layer when one exists. The paper notes that "the input and head biases are copied: $\tilde{b}_{\text{in}}^\ell = b_{\text{in}}^\ell, \tilde{b}_{\text{head}} = b_{\text{head}}$" (Theorem 1).

After these absorptions, every LayerNorm in the network has been reduced to a pure RMSNorm operation (times a constant $\sqrt{D}$), and all the linear operations that LayerNorm performed are now folded into adjacent weight matrices. This conversion is exact — it does not change any numerical output of the network. It is purely an algebraic reordering that sets the stage for the invariance theorem.

Why this conversion matters: RMSNorm has the crucial property that its operation commutes with orthogonal transformations in a specific way: multiplying the input by an orthogonal matrix $Q$, applying RMSNorm, and then multiplying by $Q^\top$ produces exactly the same result as applying RMSNorm to the original input. LayerNorm does not have this property because the mean-subtraction breaks the orthogonality relationship. By absorbing the mean-subtraction into the weight matrices, the paper converts the network into a form where the invariance can be exploited.

The conversion is described as "a straightforward change in the order of operations" (Section 3.2) because it simply reassociates matrix multiplications that are already being performed — no new operations are added to the forward pass, and no numerical approximation is involved.

Computational Invariance: Theorem 1

This is the central theoretical result of the paper. Theorem 1 states that for an RMSNorm-connected transformer network, applying an orthogonal transformation $Q$ (where $Q$ is a $D \times D$ matrix satisfying $Q^\top Q = Q Q^\top = I$) to the embedding matrix, and then absorbing $Q$ and $Q^\top$ into every subsequent weight matrix according to specific rules, produces a network that computes exactly the same outputs as the original on every input.

Formally, the transformed network is defined by:

$\tilde{W}_{\text{embd}} = W_{\text{embd}} Q$

$\tilde{W}_{\text{in}}^\ell = Q^\top W_{\text{in}}^\ell$

$\tilde{W}_{\text{out}}^\ell = W_{\text{out}}^\ell Q$

$\tilde{b}_{\text{out}}^\ell = Q^\top b_{\text{out}}^\ell$

$\tilde{W}_{\text{head}} = Q^\top W_{\text{head}}$

with $\tilde{b}_{\text{in}}^\ell = b_{\text{in}}^\ell$ and $\tilde{b}_{\text{head}} = b_{\text{head}}$ (input and head biases are copied unchanged).

What the transformations compute, operationally:

• Equation (3), $\tilde{W}_{\text{embd}} = W_{\text{embd}} Q$: The embedding lookup is post-multiplied by $Q$. Since $W_{\text{embd}}$ maps token indices to $D$-dimensional vectors, this rotates every embedding vector by the same orthogonal transformation. The initial signal $\tilde{X} = X Q$ is simply the original signal $X$ expressed in a rotated coordinate system.

• Equation (4), $\tilde{W}_{\text{in}}^\ell = Q^\top W_{\text{in}}^\ell$: The input weight matrix of each block is pre-multiplied by $Q^\top$. Since the incoming signal to this block is $\tilde{X} = XQ$ (due to the previous transformation steps), the product is $\tilde{X} \tilde{W}_{\text{in}}^\ell = XQ Q^\top W_{\text{in}}^\ell = X W_{\text{in}}^\ell$. The $Q$ from the signal and the $Q^\top$ from the weight matrix cancel, so the input to the block's nonlinearity $\sigma$ is unchanged from the original network.

• Equation (5), $\tilde{W}_{\text{out}}^\ell = W_{\text{out}}^\ell Q$: The output weight matrix is post-multiplied by $Q$. After the nonlinearity $\sigma$ produces some intermediate result $Z'$ (same as in the original network due to the cancellation above), the product is $Z' \tilde{W}_{\text{out}}^\ell = Z' W_{\text{out}}^\ell Q = Z Q$, where $Z$ is the block's output in the original network. So the block's output is the original output rotated by $Q$.

• Equation (6), $\tilde{b}_{\text{out}}^\ell = Q^\top b_{\text{out}}^\ell$: The output bias is rotated to be consistent with the rotated output weight matrix, ensuring the bias addition produces the same final vector.

• Equation (7), $\tilde{W}_{\text{head}} = Q^\top W_{\text{head}}$: At the final layer, the signal arriving at the head is $\tilde{X} = XQ$ (accumulated from all previous rotations). Multiplying by $\tilde{W}_{\text{head}}$ gives $\tilde{X} \tilde{W}_{\text{head}} = XQ Q^\top W_{\text{head}} = X W_{\text{head}}$, so the logits are identical to the original network.

The crucial step — why RMSNorm makes this work: The proof hinges on line 2 and line 5 of Algorithm 1 (the forward pass). After the embedding rotation produces $\tilde{X} = XQ$, RMSNorm is applied. The paper proves in Appendix A.1 (Equation 2 / Equation 10) that:

$\text{RMSNorm}(XQ) Q^\top = \text{RMSNorm}(X)$

This holds because RMSNorm divides each row by its Euclidean norm, and orthogonal transformations preserve Euclidean norms: $\|xQ\| = \|x\|$. After RMSNorm divides by the norm, the remaining $Q^\top$ rotates the normalized vector back to the original orientation. LayerNorm would not satisfy this relationship because its mean-subtraction step is not norm-preserving under rotation — the mean of $xQ$ is not simply the mean of $x$ rotated, so the normalization is different.

On line 5 of Algorithm 1, the residual connection adds the block's input $\tilde{X}$ to the block's output $\tilde{Z}$, both of which are in the rotated frame (both have the factor $Q$ on the right). Their sum is $(X + Z)Q$, so when RMSNorm is applied next, the same commutation property preserves the equivalence.

Why this form — what alternatives would fail: The theorem requires that the same orthogonal matrix $Q$ be used for the pre-multiplication of input matrices (Equation 4) and the post-multiplication of output matrices (Equation 5), with the latter using $Q$ and the former using $Q^\top$. If different matrices were used for input and output within the same block, the cancellation $Q Q^\top = I$ would not occur, and the nonlinearity $\sigma$ would receive different inputs. The theorem also requires that $Q$ be orthogonal (not just any invertible matrix) because the RMSNorm commutation property relies on norm preservation — a non-orthogonal transformation would change the norm of each row, causing RMSNorm to divide by a different value and breaking the equivalence.

The computational invariance is what the paper calls a "degree of freedom" — the network's computation is invariant to the choice of $Q$, meaning $Q$ can be chosen arbitrarily without affecting the model's output. This freedom is what SliceGPT exploits: rather than choosing $Q$ arbitrarily, the method chooses $Q$ based on PCA of the activations, so that the rotated representation concentrates information into leading dimensions.

Extension to Per-Block Transformations

The single-$Q$ version of Theorem 1 rotates the entire network by the same orthogonal matrix. In practice, the signal distributions at different layers of a transformer are different — early layers process low-level token representations while later layers encode higher-level semantic information. The principal components of the signal at layer 3 are not the same as those at layer 27. Using a single global rotation would mean that the PCA basis at one layer is suboptimal for another, limiting how much slicing can be done before accuracy degrades.

The paper therefore extends the invariance to allow a different orthogonal matrix per block, denoted $Q_\ell$ for block $\ell$. The transformation rules are nearly identical to Theorem 1, but with $Q$ replaced by $Q_\ell$ at each block:

• Input matrix: $\tilde{W}_{\text{in}}^\ell = Q_\ell^\top W_{\text{in}}^\ell$
• Output matrix: $\tilde{W}_{\text{out}}^\ell = W_{\text{out}}^\ell Q_\ell$
• Output bias: $\tilde{b}_{\text{out}}^\ell = Q_\ell^\top b_{\text{out}}^\ell$

The embeddings use $Q_0$ (the rotation for the first RMSNorm): $\tilde{W}_{\text{embd}} = W_{\text{embd}} Q_0$. The head uses $Q_L$ (the rotation for the last block): $\tilde{W}_{\text{head}} = Q_L^\top W_{\text{head}}$.

The residual connection problem and its solution: The extension to per-block $Q_\ell$ introduces a new complication that does not arise in the single-$Q$ case. On line 5 of Algorithm 1, the residual connection adds the input to the block (which has been rotated by $Q_{\ell-1}$ from the previous block's output transformation) to the output of the current block (which has been rotated by $Q_\ell$). These two quantities are in different rotated coordinate systems, so they cannot be directly added.

The solution is to insert a linear transformation into the residual connection: multiply the residual path by $Q_{\ell-1}^\top Q_\ell$. This transforms the residual from the $Q_{\ell-1}$-rotated frame to the $Q_\ell$-rotated frame, so that both terms in the addition are expressed in the same basis. The transformed residual connection becomes:

$\tilde{X} \leftarrow Q_{\ell-1}^\top Q_\ell \tilde{X} + \tilde{Z}$

where $\tilde{X}$ is the input to the block (in the $Q_{\ell-1}$ frame) and $\tilde{Z}$ is the output of the block (in the $Q_\ell$ frame).

Cost of the per-block extension: Unlike the weight matrix transformations — which are pre-computed and fused into the stored model weights — these residual connection matrices $Q_{\ell-1}^\top Q_\ell$ must be applied at inference time. Each is a $D \times D$ matrix multiplication applied to the $N \times D$ signal at every residual connection. The paper acknowledges this adds "a small ($D \times D$) overhead to the model" (Section 3.3). However, after slicing (Section 3.4), these matrices are themselves reduced in size to $D_{\text{small}} \times D_{\text{small}}$, and the paper's empirical speedup results (Table 2, Appendix A.6) demonstrate that the overall inference time nevertheless decreases.

Why per-block transformations are necessary: The paper states this motivation crisply: "We quickly saw that the signals at different blocks of the network were not aligned, and that we would need to apply a different orthogonal matrix at each block" (Section 3.3). The spectrum analysis in Appendix A.4 (Figure 8) confirms this — the eigenvalue distributions of the signal covariance matrices differ across layers, with early layers showing faster decay (more compressible) than later layers. Using a single global $Q$ would mean choosing a rotation that is a compromise across all layers, failing to capture the layer-specific directions of variance that PCA identifies.

PCA-Based Computation of $Q_\ell$

With the per-block invariance established, the paper needs a principled way to choose each $Q_\ell$. The goal is to find a rotation that concentrates the information in the signal matrix $X_\ell$ into as few dimensions as possible, so that when dimensions are later deleted, the reconstruction error — the amount of signal information lost — is minimized. This is exactly what Principal Component Analysis provides.

Covariance matrix construction: For each block $\ell$, the paper collects the output of the $\ell^{\text{th}}$ RMSNorm block for every sequence in a calibration dataset. Let $X_{\ell,i}$ be the $N_i \times D$ signal matrix at block $\ell$ for the $i^{\text{th}}$ calibration sequence (where $N_i$ is the sequence length of that sample). The covariance matrix is computed as:

$C_\ell = \sum_i X_{\ell,i}^\top X_{\ell,i}$

where $X_{\ell,i}^\top X_{\ell,i}$ is a $D \times D$ matrix representing the uncentered covariance of the activations at block $\ell$ for sequence $i$ (uncentered because PCA is performed without mean-subtraction — the paper notes that mean subtraction has already been absorbed into the weight matrices during the LayerNorm conversion, so the signals are already approximately zero-mean). The summation over $i$ accumulates covariance across all calibration sequences.

Eigendecomposition: The orthogonal matrix $Q_\ell$ is set to the eigenvectors of $C_\ell$, sorted by decreasing eigenvalue. If $C_\ell = V \Lambda V^\top$ is the eigendecomposition (with $\Lambda$ diagonal and $V$ orthogonal), then $Q_\ell = V$. The eigenvectors form an orthonormal basis for the $D$-dimensional signal space, and the corresponding eigenvalues are the variances of the signal along each eigenvector direction. Sorting by decreasing eigenvalue means the first column of $Q_\ell$ is the direction of maximum variance, the second column is the direction of second-most variance (orthogonal to the first), and so on.

Operational meaning: When the signal $X_\ell$ is multiplied by $Q_\ell$, the resulting columns of $X_\ell Q_\ell$ are the projections of the signal onto the principal components. The first column (projection onto the first eigenvector) has the largest variance; the last column has the smallest. This means that after rotation, the "importance" of each dimension — measured by how much the signal varies along that dimension — decreases monotonically from left to right.

Calibration dataset design: The paper uses two calibration sets: WikiText-2 training data and the Alpaca training dataset (Section 4). The ablation in Appendix A.3 (Figure 7) examines the sensitivity to calibration set size and sequence length. Key findings:

• Calibration set size: At least 128 samples are needed for stable PCA; beyond this, improvements are marginal. The main experiments use 1024 samples.
• Sequence length: Longer sequences (up to 4096) produce better perplexity because they provide more signal vectors per sample ($N_i \times B$ total vectors, where $B$ is the number of samples). The main experiments use sequence length 2048.
• Interaction: Given a fixed total number of embedding vectors $NB$, having larger $N$ (longer sequences) is preferable to larger $B$ (more sequences), likely because longer sequences expose more diverse in-context signal patterns.

Numerical precision: The PCA computation uses double precision (FP64) for eigenvector calculation. The paper explicitly warns that using single precision (FP32) leads to accuracy degradation: Table 4 in Appendix A.2 shows that for LLAMA-2 70B at 30% slicing, FP32 PCA yields perplexity 8.75 compared to 5.05 with FP64 PCA (both compared to 3.32 dense). The degradation is most severe for larger models, suggesting that the eigenvalue computation becomes ill-conditioned as the embedding dimension grows and numerical errors in the eigenvectors accumulate.

Why PCA — what property does it optimize?: PCA minimizes the L2 reconstruction error when dimensions are discarded. Specifically, if we keep only the first $D_{\text{small}}$ columns of $Q_\ell$ (the leading eigenvectors) and discard the rest, the reconstruction $\tilde{X}_\ell = X_\ell Q_\ell D D^\top Q_\ell^\top$ (where $D$ selects the first $D_{\text{small}}$ columns) minimizes $\|X_\ell - \tilde{X}_\ell\|^2$ among all linear dimension-reduction mappings to $D_{\text{small}}$ dimensions. This is the $L_2$-optimal sense referred to in Section 3.4.

The sequential computation issue: Because different blocks get different $Q_\ell$, the signal distribution at block $\ell + 1$ depends on all the previous $Q_1, \ldots, Q_\ell$. The paper handles this by computing $Q_\ell$ sequentially: first transform blocks 1 through $\ell - 1$ using their already-computed $Q$ matrices, then run calibration data through the partially-transformed network to collect $X_\ell$, compute $C_\ell$, and set $Q_\ell$. The paper states: "We use the output of the transformed network to calculate the orthogonal matrices of the next layers" (Section 3.3). This ensures that the PCA at each layer reflects the actual signal distribution that will occur in the final transformed network, not the distribution in the original network.

Slicing: Deleting Rows and Columns

After all $Q_\ell$ have been computed and absorbed into the weight matrices, the network is expressed in a rotated basis where the signal dimensions are ordered by decreasing variance. The final step is to actually delete the low-variance dimensions.

The deletion matrix $D$: The paper defines a $D \times D_{\text{small}}$ deletion matrix which contains $D_{\text{small}}$ columns of the $D \times D$ identity matrix. Typically, $D$ selects the first $D_{\text{small}}$ columns (corresponding to the largest eigenvalues), so $D = [I_{D_{\text{small}}}; 0]^\top$ (the top $D_{\text{small}}$ rows of the identity matrix, transposed). The slicing percentage determines $D_{\text{small}}$: for 25% slicing, $D_{\text{small}} = 0.75 \times D$; for 30% slicing, $D_{\text{small}} = 0.70 \times D$.

Which matrices get sliced, and how: The slicing is applied uniformly to every component that interacts with the embedding dimension $D$:

1. Input weight matrices: The transformed input matrix $\tilde{W}_{\text{in}}^\ell = Q_\ell^\top W_{\text{in}}^\ell$ has shape $D \times (kD)$ for attention (where $k$ accounts for key, query, value concatenation) or $D \times D_{\text{ff}}$ for FFN. Slicing removes the bottom rows of this matrix — specifically, the rows corresponding to the eigenvectors with smallest eigenvalues. The sliced input matrix is $D^\top \tilde{W}_{\text{in}}^\ell$, which has shape $D_{\text{small}} \times (kD)$ or $D_{\text{small}} \times D_{\text{ff}}$. This means the block's nonlinearity now receives a signal with only $D_{\text{small}}$ dimensions.

2. Output weight matrices: The transformed output matrix $\tilde{W}_{\text{out}}^\ell = W_{\text{out}}^\ell Q_\ell$ has shape $(kD) \times D$ or $D_{\text{ff}} \times D$. Slicing removes the rightmost columns — the columns corresponding to the small-eigenvalue directions. The sliced output matrix is $\tilde{W}_{\text{out}}^\ell D$, which has shape $(kD) \times D_{\text{small}}$ or $D_{\text{ff}} \times D_{\text{small}}$. This produces an output signal in the $D_{\text{small}}$-dimensional space.

3. Embedding matrix: The transformed embedding $\tilde{W}_{\text{embd}} = W_{\text{embd}} Q_0$ has shape $V \times D$ (where $V$ is vocabulary size). Slicing removes the rightmost columns, yielding shape $V \times D_{\text{small}}$. The network's initial signal is now $D_{\text{small}}$-dimensional.

4. Head matrix: The transformed head $\tilde{W}_{\text{head}} = Q_L^\top W_{\text{head}}$ has shape $D \times V$. Slicing removes the bottom rows, yielding shape $D_{\text{small}} \times V$. The logits are computed from the $D_{\text{small}}$-dimensional final signal.

5. Residual connection matrices: The matrix $Q_{\ell-1}^\top Q_\ell$ (shape $D \times D$) is sliced on both rows and columns: $D^\top (Q_{\ell-1}^\top Q_\ell) D$, yielding a $D_{\text{small}} \times D_{\text{small}}$ matrix. This is the only component where both dimensions shrink.

6. Bias terms: Output biases $\tilde{b}_{\text{out}}^\ell$ (shape $D$) become $D^\top \tilde{b}_{\text{out}}^\ell$ (shape $D_{\text{small}}$) by keeping only the first $D_{\text{small}}$ components.

Parameter count reduction: For a weight matrix originally of shape $D \times D$, after transformation and slicing it becomes $D_{\text{small}} \times D$ (input side) or $D \times D_{\text{small}}$ (output side). The reduction factor is $D_{\text{small}} / D = 1 - \text{slicing\_percentage}$. At 25% slicing, each such matrix has 75% of its original rows or columns. However, since the embedding dimension propagates through the entire network, the total parameter reduction exceeds the per-matrix reduction because both input and output dimensions of all subsequent layers shrink.

Why the slicing pattern is column/row deletion rather than arbitrary sparsity: The paper emphasizes that the result of slicing is "smaller (dense) matrices" (Section 3.4), not sparse matrices. After slicing, the weight matrices are still stored and operated on as dense arrays — they are just physically smaller. The matrix multiplication $\tilde{X} \tilde{W}$ uses standard dense GEMM kernels with no sparsity overhead. This is the key architectural advantage over 2:4 sparsity (which requires special kernel support) and unstructured sparsity (which sees limited speedup in practice).

Why this slicing is valid — the PCA guarantee: The slicing deletes the dimensions of $X_\ell Q_\ell$ corresponding to the smallest eigenvalues of $C_\ell$. PCA theory guarantees that this minimizes the Frobenius norm of the reconstruction error $\|X_\ell - \tilde{X}_\ell\|_F$ for a given target dimensionality $D_{\text{small}}$. However, the paper's approach is not a standard PCA reconstruction — instead of reconstructing the full-dimensional signal and passing it to the next layer, the sliced network simply passes the lower-dimensional signal directly. This works because the downstream weight matrices have been correspondingly sliced: the next block's input matrix expects a $D_{\text{small}}$-dimensional input (having had its rows sliced), so no up-projection back to $D$ dimensions is needed. The network effectively operates natively in the reduced dimension.

Constant vs. variable slicing across layers: The main experiments apply a constant slicing percentage to all layers. However, Appendix A.4 (Table 6) explores variable slicing, where the percentage is determined by the eigenvalue spectrum at each layer: instead of specifying a fixed percentage, the method sets a threshold on the fraction of total variance to discard, and deletes however many dimensions fall below that threshold. The results are mixed: variable slicing improves WikiText-2 perplexity for OPT models (e.g., OPT 6.7B improves from 12.10 to 11.94 perplexity at ~25% total slicing) but degrades it for LLAMA-2 models (LLAMA-2 7B degrades from 6.84 to 7.63). The paper attributes this to differences in the eigenvalue distributions (Figure 8): LLAMA-2 models have more tightly compressed spectra without clearly dominant principal components, making uniform slicing a more robust strategy.

The full pipeline in sequence: Putting all five components together, the complete SliceGPT compression of a pre-trained transformer proceeds as:

1. Load the pre-trained model and identify all LayerNorm instances, linear layers, and residual connections.
2. Convert LayerNorm to RMSNorm by absorbing $M$ into output matrices, $\text{diag}(\alpha)$ into input matrices, and handling embeddings and head analogously. The network now computes the same function with pure RMSNorm normalization.
3. Select a calibration dataset (typically 1024 sequences of length 2048 from WikiText-2 or Alpaca).
4. For each block $\ell = 1, \ldots, L$: Run the calibration data through blocks 1 to $\ell-1$ (already transformed) and collect $X_\ell$ — the output of the $\ell^{\text{th}}$ RMSNorm. Compute $C_\ell = \sum_i X_{\ell,i}^\top X_{\ell,i}$. Perform eigendecomposition in double precision to obtain $Q_\ell$. Absorb $Q_\ell$ into $W_{\text{in}}^\ell$, $W_{\text{out}}^\ell$, $b_{\text{out}}^\ell$, and insert $Q_{\ell-1}^\top Q_\ell$ into the residual connection.
5. After all blocks are transformed, apply $Q_0$ to $W_{\text{embd}}$ and $Q_L^\top$ to $W_{\text{head}}$.
6. Slice: For each affected matrix, multiply by the deletion matrix $D$ (or $D^\top$) on the appropriate side to remove the rows/columns corresponding to the smallest principal components. The slicing percentage determines $D_{\text{small}}$ and which rows/columns are retained.
7. Output the compressed model as a standard dense PyTorch model with all embedding dimensions reduced from $D$ to $D_{\text{small}}$.

The result is a model that is structurally identical to the original — same number of layers, same attention heads, same nonlinearities — but with every weight matrix smaller and every inter-block signal shorter. The forward pass executes exactly the same sequence of operations as the original model, just on smaller tensors.

---

Summary of Design Choices and Justifications

• RMSNorm conversion over direct LayerNorm handling: The computational invariance theorem requires RMSNorm because the commutation property (Equation 2) depends on norm preservation under orthogonal transformations. LayerNorm's mean subtraction breaks this property. Converting LayerNorm to RMSNorm by absorbing linear components is an exact algebraic rewrite, so this choice costs nothing in accuracy while enabling the entire subsequent pipeline.

• Orthogonal transformations over general invertible matrices: Orthogonal matrices preserve vector norms, which is what makes RMSNorm(XQ)Q$^\top$ = RMSNorm(X) hold. A general invertible matrix would change the norm of each row, causing RMSNorm to divide by a different value and producing a different output. Orthogonality also means Q$^{-1}$ = Q$^\top$, simplifying the absorption rules (you only need Q and Q$^\top$, not separate forward and inverse matrices).

• Per-block Q$_\ell$ over a single global Q: Signal distributions differ across layers (Appendix A.4, Figure 8), so a single rotation optimized for the average distribution would be suboptimal everywhere. Per-block rotations capture layer-specific principal components, enabling more aggressive slicing before accuracy degrades. The cost is the residual connection matrices Q$_{\ell-1}^\top$Q$_\ell$, but these are small relative to the weight matrix savings.

• PCA over other dimensionality reduction methods: PCA is optimal in the L2 reconstruction sense and produces orthogonal transformations (which is required for the invariance). Alternative dimensionality reduction methods like random projections or autoencoders would not produce orthogonal matrices satisfying Q$^\top$Q = I, breaking the invariance proof.

• Double-precision eigendecomposition: The paper empirically found that FP32 eigenvalue computation leads to numerical errors that compound in larger models (Table 4), causing significant perplexity degradation. FP64 avoids this at the cost of higher memory usage during the one-time PCA computation (which runs on a single GPU in 1-3 hours).

• Calibration dataset size of ~1024 sequences with length 2048: The ablation in Appendix A.3 shows that at least 128 samples are needed, and longer sequences are more informative than more sequences for a fixed total token count. The 1024 × 2048 configuration provides ~2M embedding vectors per layer, which is sufficient to estimate the $D \times D$ covariance matrix (where $D$ ranges from 4096 for LLAMA-2 7B to 8192 for LLAMA-2 70B).

• Uniform slicing percentage across layers (default): While variable slicing (Appendix A.4, Table 6) can improve OPT models, it degrades LLAMA-2 models. The paper's main results therefore use a constant percentage, which is simpler to specify and more robust across model families. The spectrum analysis suggests this is because LLAMA-2's eigenvalue decay is more uniform across layers, making a constant percentage a reasonable approximation.

• Slicing via deletion matrix $D$ rather than retraining: The deletion matrix approach is a zero-shot operation — no gradient steps, no fine-tuning, no additional data beyond the calibration set used for PCA. This is what enables the 1-3 hour compression time on a single GPU. Alternative approaches like pruning with recovery fine-tuning (as in LLM-Pruner) or low-rank factorization with fine-tuning require substantially more compute.

4. Key Insights and Innovations

Innovation 1: Computational Invariance as a New Axis of Model-Surgical Freedom

The paper's deepest conceptual contribution is the identification and exploitation of computational invariance — the observation that transformer networks with RMSNorm connections possess an exact algebraic symmetry under orthogonal transformations of their weight matrices. This is not an approximation, a compression heuristic, or a training trick. It is a degree of freedom that was always present in the architecture but never previously recognized or exploited for compression.

What makes this intellectually distinctive is the type of insight it represents. Prior compression work — whether pruning (Han et al., 2016; Frantar & Alistarh, 2023), quantization (Frantar et al., 2022; Dettmers et al., 2022), low-rank factorization (Hu et al., 2021; Noach & Goldberg, 2020), or structural pruning (Ma et al., 2023a) — operates on the assumption that compressing a trained model means approximating its weights or activations while accepting some degradation in output quality. The tradeoff is always accuracy vs. compression ratio, managed through careful engineering. SliceGPT's invariance theorem (Theorem 1) changes the game by showing that certain transformations produce zero degradation — the network computes exactly the same function before and after the orthogonal rotations are applied.

This is a fundamental shift, not an incremental refinement. The invariance reveals that a pre-trained transformer's weight configuration is not a single point in parameter space but an equivalence class under orthogonal transformations: for any orthogonal matrix $Q$, the transformed network $\tilde{W} = f(W, Q)$ computes the identical function. The network you downloaded from HuggingFace is just one representative of an infinite family of equivalent networks — one that happens to be expressed in whatever coordinate system the training process converged to. SliceGPT's key move is to choose a better representative from this equivalence class — specifically, the one aligned with the PCA basis of the activations — such that when dimensions are subsequently deleted, the information loss is minimized.

This framing is genuinely novel. It draws on a concept familiar from differential geometry and physics (gauge invariance, coordinate-free representations) but applies it to a concrete engineering problem in deep learning. The parallel to gauge theories is striking: just as the laws of electromagnetism are invariant under local phase transformations of the wavefunction, the computation of an RMSNorm-connected transformer is invariant under orthogonal rotations of the embedding space. SliceGPT is, in effect, performing gauge-fixing — choosing a specific gauge (rotation) that makes subsequent truncation maximally harmless.

Prior work had no analog to this. SparseGPT (Frantar & Alistarh, 2023) compensates for weight removal by updating the remaining weights using Hessian information — but this is a local, approximate compensation, not an exact invariance. The paper's own baseline experiment — naively deleting columns with the smallest norm — produced perplexity in the thousands (Section 4), confirming that without the invariance theorem to justify the rotation, arbitrary dimension deletion is catastrophic. The invariance is what transforms column deletion from a destructive operation into a principled compression.

Evidence anchors: The proof in Theorem 1 and Appendix A.1 is mathematically exact, not empirical. The experimental confirmation that the invariance holds in practice is implicit in the entire results table — if the invariance were approximate rather than exact, even 10% slicing would show some degradation beyond what PCA reconstruction error predicts, but Table 1 shows that 10% sliced OPT 66B achieves 9.43 perplexity vs. 9.33 dense, consistent with near-lossless rotation and minimal slicing error.

Why it matters beyond this paper: The computational invariance insight opens a broader research direction that the paper only begins to explore. Any operation that can be expressed as a linear transformation applied to the signal between RMSNorm blocks — not just PCA rotation — can potentially be absorbed into the weight matrices without changing the model. This could include transformations that improve quantization (by rotating to a basis with more uniform value ranges), that enhance hardware utilization (by aligning computation with memory access patterns), or that facilitate model merging (by aligning the coordinate systems of independently trained models). The paper explicitly gestures toward this in Section 5: "We hope that our observation of computational invariance can help future research in improving the efficiency of deep learning models, and perhaps inspire new theoretical insights." This is not boilerplate — it's an accurate identification of a conceptual door that has been opened.

Innovation 2: Embedding Dimension Reduction as a Distinct Compression Axis

SliceGPT introduces a compression paradigm that is categorically different from the four canonical approaches (distillation, tensor decomposition, pruning, quantization). Rather than approximating weights, sparsifying matrices, or reducing bit precision, SliceGPT changes the architecture's fundamental width parameter $D$ — the embedding dimension that propagates through every layer of the network. This is not pruning (which zeros weights but leaves matrix dimensions unchanged), not low-rank factorization (which replaces one matrix with two but keeps the signal dimension constant), and not quantization (which reduces per-element precision). It is a dimensionality reduction of the network itself.

The distinction is easiest to see by comparing Figure 1's three panels. In unstructured sparsity (left), $W$ has zeros scattered throughout but $X$ is dense and full-sized. In 2:4 sparsity (middle), $W$ has a regular zero pattern but $X$ is still dense and full-sized. In SliceGPT (right), both $W$ and $X$ are physically smaller — the sparsity has been concentrated and then removed entirely, producing a network that operates natively in a lower-dimensional space. The signal $X$ is no longer $N \times D$; it's $N \times D_{\text{small}}$. Every subsequent matrix multiplication is smaller by the same factor.

Why is this a conceptual advance rather than just a different tradeoff point? Because it compounds in a way that weight-only sparsity cannot. Consider a 25% reduction in $D$:

• The embedding matrix shrinks from $V \times D$ to $V \times 0.75D$ (25% reduction in parameters and memory for embedding lookups).
• The signal $X$ at every layer is 25% smaller, reducing activation memory by 25% for all intermediate tensors. This is critical for GPU memory pressure, since activations often dominate memory usage during inference (especially with large batch sizes).
• Weight matrices that were $D \times D$ become $0.75D \times D$ (input side) or $D \times 0.75D$ (output side), a 25% reduction per matrix. But because both dimensions shrink, the next block's input matrix goes from $D \times D$ to $0.75D \times 0.75D$ after its rows are also sliced — a cumulative ~44% reduction in that matrix's total parameters.
• The residual connection matrices $Q_{\ell-1}^\top Q_\ell$ shrink in both dimensions: from $D \times D$ to $0.75D \times 0.75D$, a ~44% reduction.

This compounding effect is visible in the practical results but is an architectural property, not an empirical accident. SparseGPT 2:4 with the same nominal "50% sparsity" still requires the same number of GPUs as the dense model (Appendix A.6 — LLAMA-2 70B uses 2 H100s for both dense and 2:4 sparse), because the activations and the matrix shapes haven't changed, and the weight tensors still occupy the same memory footprint (the sparsity pattern requires metadata storage). SliceGPT at 50% reduces LLAMA-2 70B from 2 GPUs to 1 (Table 11) — a factor-of-2 reduction in hardware requirements that 2:4 sparsity cannot provide regardless of its theoretical FLOP reduction.

Prior structural pruning methods for CNNs (ThiNet, channel pruning) achieved similar dimensional reduction, but the paper's contribution is making this possible for transformers without retraining. The reason channel pruning works in CNNs is that batch normalization layers provide a natural per-channel importance signal. Transformers lack this — LayerNorm operates per-token, not per-channel in the same way — and the tight coupling through residual connections and multi-head attention makes arbitrary dimension removal catastrophic (as the paper's norm-based pruning baseline confirms). The computational invariance theorem is what unlocks embedding dimension reduction specifically for transformers, and the per-block PCA extension is what makes it practical across all layers.

Evidence anchors: The GPU count reduction in Table 2 is the cleanest demonstration. LLAMA-2 70B goes from 4 A100 GPUs (dense) to 3 GPUs (25% sliced). On consumer GPUs, it goes from 7 to 5. The total GPUms drops from 500 to 330 (A100) and from 1764 to 1075 (RTX6000). These are wall-clock, end-to-end measurements that include the residual connection overhead — the dimensional reduction's benefits survive all overhead costs. For throughput (Table 11), 50% sliced LLAMA-2 70B achieves $3.75\times$ the throughput of the dense model when using the same number of GPUs (since it fits larger batch sizes), or equivalently achieves similar throughput on 1 GPU that the dense model achieves on 2 — a $2\times$ hardware efficiency gain.

Innovation 3: PCA-Guided Rotation as a Principled Criterion for Structured Deletion

The computational invariance theorem says that any orthogonal $Q$ produces an equivalent network. This freedom is powerful but raises a critical question: which $Q$ should you choose? The paper's answer — compute $Q$ as the PCA basis of the activations at each block — transforms an arbitrary degree of freedom into a principled compression strategy. This connection between algebraic invariance and statistical optimality is the paper's third conceptual contribution.

The logic is elegant: PCA finds the orthogonal basis that diagonalizes the activation covariance matrix $C_\ell = X_\ell^\top X_\ell$, ordering dimensions by the variance they carry. In this basis, the "energy" of the signal (its total variance) is concentrated into the leading dimensions, with trailing dimensions contributing negligibly. When the deletion matrix $D$ removes these trailing dimensions, PCA guarantees that the L2 reconstruction error $\|X_\ell - \tilde{X}_\ell\|$ is minimized among all linear projections to $D_{\text{small}}$ dimensions. The network is not being arbitrarily truncated — it's being truncated along the directions that, statistically, carry the least information about the activations on the calibration data.

What makes this distinctive relative to prior work is that it replaces heuristic importance measures with a statistically grounded one. Pruning methods must decide which weights or structures to remove, and this decision rule — the "saliency criterion" — largely determines the quality of the compressed model. Prior criteria include:

• Magnitude (Han et al., 2016): remove weights with smallest absolute values. Simple but often removes important weights that happen to be small (e.g., weights that are part of a high-precision cancellation).
• Hessian-based (Hassibi et al., 1993; Frantar & Alistarh, 2023): remove weights whose deletion minimally impacts the loss, approximated via second-order Taylor expansion. More principled but requires computing or approximating the inverse Hessian.
• Norm-based structured pruning (the paper's own baseline): remove filters or channels with smallest L2 norm. The paper found this catastrophically ineffective for transformers (perplexity > 1000).
• Regularization-based (Liu et al., 2017): add sparsity-inducing regularization to batch-norm scaling factors during training, then prune channels with near-zero factors. Requires training-time modification.

SliceGPT's innovation is that the PCA criterion is simultaneously principled and cheap. It is principled because it minimizes reconstruction error in the L2 sense — a well-understood optimality property. It is cheap because it requires only a forward pass on calibration data (no gradients, no Hessian, no retraining). And it is architecture-appropriate because the orthogonal transformation that PCA produces is exactly the type of transformation that the invariance theorem can absorb — PCA doesn't just tell you which dimensions to delete, it provides the $Q$ that makes deletion possible in the first place.

This is a non-trivial alignment. Many dimensionality reduction methods exist (autoencoders, random projections, independent component analysis, non-negative matrix factorization), but only PCA produces an orthogonal transformation matrix ordered by variance. If you used an autoencoder, the encoder would not be orthogonal, the invariance theorem would not apply, and the transformation couldn't be absorbed into the weight matrices. PCA is not just one choice among many — it's the unique choice that satisfies both the mathematical requirements of the invariance theorem (orthogonality) and the statistical requirement of minimizing information loss under dimension reduction.

The paper's spectrum analysis (Appendix A.4, Figure 8) provides empirical grounding for why this works well for OPT but less well for LLAMA-2. The eigenvalue distributions reveal that OPT models have more rapid spectral decay — the leading principal components capture a larger fraction of total variance — making them more compressible via PCA-guided slicing. LLAMA-2 models have tighter spectra, meaning variance is more evenly distributed across dimensions, so deleting any fixed percentage of dimensions loses more information. This explains the consistent pattern in Table 1 where SliceGPT achieves better perplexity on OPT than LLAMA-2 at the same slicing percentage, and it demonstrates that the PCA criterion is doing meaningful work: when the spectrum decays quickly (the signal is low-rank), PCA finds a good low-dimensional representation; when the spectrum is flat (the signal is high-rank), PCA cannot concentrate information and slicing hurts more.

Evidence anchors: Table 1 shows the difficulty-dependent pattern clearly. At 25% slicing, OPT 66B goes from 9.33 to 9.68 perplexity (a 0.35-point increase), while LLAMA-2 70B goes from 3.32 to 4.60 (a 1.28-point increase). The absolute perplexity numbers are not directly comparable across model families, but the relative degradation is: OPT 66B loses ~3.7% of its dense perplexity, while LLAMA-2 70B loses ~38.6%. This is explained by Figure 8: OPT's spectra decay faster, so 25% dimension deletion removes less total variance.

Why this matters beyond this paper: The PCA-guided rotation principle decouples the choice of transformation from the choice of what to delete. Future work could explore alternative transformation criteria that satisfy the orthogonality constraint while optimizing for different objectives: maximizing quantization friendliness (by rotating to a basis with uniform value ranges), enhancing sparsity of the resulting weight matrices (by rotating to a basis where weights concentrate in few large entries), or improving hardware utilization (by aligning the computational basis with memory layout). The invariance theorem provides the algebraic framework; PCA is one particularly useful instantiation of it.

Innovation 4: The Diagnostic Role of Spectral Analysis in Compression

A less obvious but intellectually significant contribution is the paper's use of eigenvalue spectrum analysis not just as a computational tool but as a diagnostic instrument for understanding when and why compression will work. This transforms compression from a trial-and-error empirical exercise into something with predictive structure.

The paper observes — and visualizes in Figure 8 — that the eigenvalue spectra of activation covariance matrices differ qualitatively across model families and across layers within the same model. OPT models show faster spectral decay (variance concentrated in fewer dimensions), especially in early layers. LLAMA-2 models show more uniform spectra (variance spread across many dimensions). This single observation explains nearly all the cross-model variation in SliceGPT's effectiveness:

• Why OPT compresses better than LLAMA-2 (Table 1, Figure 5): OPT's spectra are more compressible.
• Why larger models compress better than smaller ones (consistent trend in Table 1): larger models have more redundancy, manifesting as faster spectral decay.
• Why early layers are more compressible than later layers (Figure 8): the eigenvalue decay is steeper in earlier layers for both model families.
• Why variable slicing helps OPT but hurts LLAMA-2 (Table 6): OPT's layer-wise spectral variation is large enough to exploit via adaptive slicing percentages, while LLAMA-2's more uniform spectra mean variable slicing fails to identify clearly more/less compressible layers.

This diagnostic capability is a conceptual advance over prior compression work, which typically evaluates methods by reporting aggregate metrics (perplexity, accuracy) without explaining why one model compresses better than another. SparseGPT (Frantar & Alistarh, 2023) and GPTQ (Frantar et al., 2022) demonstrate that their methods work across model sizes but do not provide a spectral analysis that predicts compression difficulty a priori. The paper's spectrum analysis (Figure 8) suggests that you can estimate how compressible a model will be before applying any compression method, simply by examining the eigenvalue distribution of its activations on a small calibration set.

This has practical implications beyond SliceGPT. If a model has a flat spectrum (like LLAMA-2 smaller variants), no method that deletes dimensions will work well — the information is genuinely distributed across all dimensions. In such cases, alternative compression strategies (quantization, distillation) may be more appropriate. If a model has rapid spectral decay, dimension-reduction methods have a large margin before degradation sets in. The spectrum provides a theoretical upper bound on how much dimension reduction is possible for a given tolerance on reconstruction error.

The paper also identifies a subtlety that prevents the spectrum from being the whole story: the calibration dataset matters. Spectra computed from WikiText-2 produce different PCA bases than spectra computed from Alpaca (Figure 5, top vs. bottom rows). The downstream task performance of sliced models depends on the alignment between the calibration data distribution and the evaluation data distribution — Alpaca-calibrated models perform better on zero-shot tasks because Alpaca's instruction-following format is more similar to those benchmarks. This means the spectral analysis is not purely a property of the model; it's a property of the model-data pair, which is a more nuanced and accurate framing.

Evidence anchors: Figure 8 is the key exhibit. The log-scale normalized eigenvalue plots show visually that LLAMA-2 7B's spectrum (except layer 1) decays more slowly than OPT 6.7B's — the curves are flatter. Layer 1 of LLAMA-2 is an outlier with rapid decay, suggesting the embedding-to-first-block transition is highly structured. Table 6 quantifies the variable-slicing result: OPT models benefit from per-layer adaptive slicing (e.g., OPT 13B improves from 11.04 to 10.76 perplexity), consistent with their layer-varying spectra; LLAMA-2 models are hurt (e.g., LLAMA-2 7B degrades from 6.84 to 7.63), consistent with their uniformly flat spectra where adaptive thresholds fail to find clearly better or worse layers to differentially slice.

Why this matters as a standalone contribution: The spectral diagnostic is transferable. A practitioner considering whether to apply SliceGPT to a new model (or a new model family, or a fine-tuned variant) can first run a quick calibration pass, compute the eigenvalue spectrum, and immediately assess: (a) what slicing percentage is likely viable, (b) whether uniform or adaptive slicing is appropriate, and (c) whether dimension reduction is even the right compression strategy for this particular model. This transforms model compression from a black-box empirical exercise into a more principled engineering decision informed by measurable properties of the model's representations.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary generation benchmark is WikiText-2 (Merity et al., 2016), a standard language modeling corpus. The training split serves as the default calibration dataset for PCA computation. For zero-shot downstream evaluation, five tasks from the LM Evaluation Harness (Gao et al., 2021) are used: PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), HellaSwag (Zellers et al., 2019), ARC-e and ARC-c (Clark et al., 2018). A second calibration dataset, the Alpaca training set (Taori et al., 2023), is used for ablation experiments on downstream task transfer.

• Base model(s). Experiments span three model families: OPT (Zhang et al., 2022) at scales 125M, 1.3B, 2.7B, 6.7B, 13B, 30B, and 66B; LLAMA-2 (Touvron et al., 2023) at scales 7B, 13B, and 70B; and Phi-2 (2.8B parameters) for zero-shot task experiments. OPT 175B is excluded because it is "outperformed by smaller LLAMA-2 models." The range of scales — from 125M to 70B — tests whether compression efficacy scales with model size. The inclusion of Phi-2 probes whether small LMs can benefit from post-training slicing.

• Metrics. For the generation task, perplexity on the WikiText-2 test set is the primary metric — lower is better, and it measures how well the compressed model predicts the held-out text. For zero-shot tasks, accuracy (percentage of correct answers) is reported per task, with the mean accuracy across five tasks serving as the aggregate downstream metric. For throughput, tokens per second is measured by finding the maximum throughput via batch-size doubling until GPU memory is exhausted. For inference latency, milliseconds per token and GPUms (GPU count × ms per token) are reported.

• Baselines. The paper compares against three baselines. Dense is the original unmodified pre-trained model. SparseGPT 2:4 (Frantar & Alistarh, 2023) applies semi-structured sparsity where exactly 2 of every 4 contiguous weights are zero — this is the "only sparsity scheme which achieves speedup" on current hardware via NVIDIA sparse tensor cores. A norm-based column pruning baseline was attempted but abandoned: "we initially planned to compare our results against a scheme that pruned columns (or rows) with the smallest norm but found that this baseline was very poor, with the WikiText-2 perplexity of the model soaring into the 1000s after pruning just a few columns" (Section 4). This negative result is itself informative — it demonstrates that naive dimension deletion without SliceGPT's transformation step is catastrophic.

• Generation budget / compute accounting. The paper does not use a "generation budget" framework since SliceGPT is a one-shot compression method, not a test-time compute strategy. Instead, compression cost is measured in wall-clock time and GPU count for the PCA computation and optional recovery fine-tuning (Table 3). Inference speedup is measured in milliseconds per token (Table 2), tokens per second throughput (Table 11, Appendix A.6), and GPUms (GPU count × ms per token). The compression ratio is controlled by the slicing percentage — the fraction of embedding dimensions removed — applied uniformly across all layers (except in the variable-slicing ablation in Table 6). Slicing percentages of 10%, 20%, 25%, and 30% are evaluated, with 25% as the primary operating point for speedup experiments.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. All results are single-run evaluations of the compressed model on standard benchmarks, consistent with prior compression work (Frantar & Alistarh, 2023; Frantar et al., 2022). The calibration set is fixed (1024 sequences of length 2048 for main experiments, 128 sequences for the ablation in Table 5). For recovery fine-tuning (RFT), LoRA hyperparameters are fixed at lora_r = 32, lora_alpha = 10, sequence length 1024, with approximately 1k RFT sequences for WikiText-2 calibration and 5k for Alpaca calibration.

Main Quantitative Results

Generation Task: WikiText-2 Perplexity

Table 1 presents the core compression-quality tradeoff: WikiText-2 perplexity for OPT (seven model sizes) and LLAMA-2 (three model sizes) at slicing percentages from 10% to 30%, compared against the dense baseline and SparseGPT 2:4.

Headline numbers: For the largest models, SliceGPT at 25% slicing preserves perplexity remarkably close to dense performance. OPT 66B: dense 9.33 → SliceGPT 25% 9.68 (0.35 increase). LLAMA-2 70B: dense 3.32 → SliceGPT 25% 4.60 (1.28 increase). At 30% slicing, OPT 66B reaches 9.85 and LLAMA-2 70B reaches 5.05.

Comparison to SparseGPT 2:4: The paper emphasizes that "SparseGPT 2:4 performs worse than SliceGPT with 25% slicing in all LLAMA-2 models" (Section 4.1). Specifically, for LLAMA-2 70B: SparseGPT 2:4 achieves 4.98 vs. SliceGPT 25% at 4.60 — SliceGPT is better by 0.38 perplexity points while simultaneously achieving higher throughput (discussed below) and reducing GPU count (Table 2). For LLAMA-2 13B: SparseGPT 2:4 at 7.07 vs. SliceGPT 25% at 6.30. For LLAMA-2 7B: SparseGPT 2:4 at 8.69 vs. SliceGPT 25% at 7.24. For OPT, the pattern is similar: "30% sliced models beat 2:4 sparsity for all model sizes except 2.7B." OPT 66B: SparseGPT 2:4 at 10.22 vs. SliceGPT 30% at 9.85. The only exception is OPT 2.7B, where SparseGPT 2:4 at 14.90 edges out SliceGPT 30% at 15.83.

Scale dependence: Accuracy retention improves monotonically with model size. For OPT at 25% slicing: 125M degrades from 27.64 to 37.74 (+10.10), 1.3B from 14.61 to 17.46 (+2.85), 2.7B from 12.46 to 14.56 (+2.10), 6.7B from 10.85 to 11.90 (+1.05), 13B from 10.12 to 10.94 (+0.82), 30B from 9.56 to 10.04 (+0.48), 66B from 9.33 to 9.68 (+0.35). The absolute perplexity degradation shrinks as parameter count grows, consistent with the paper's claim that "larger models typically offer more promising opportunities for compression." For LLAMA-2 at 25% slicing: 7B degrades from 5.47 to 7.24 (+1.77), 13B from 4.88 to 6.30 (+1.42), 70B from 3.32 to 4.60 (+1.28). The relative degradation (percentage increase) follows the same decreasing trend.

Cross-family difference: OPT models consistently outperform LLAMA-2 models at equivalent slicing percentages even after accounting for their higher baseline perplexities. The paper attributes this to spectral differences (Appendix A.4, Figure 8): "SliceGPT exhibits superior performance when applied to OPT models compared to LLAMA-2 models which matches our intuition from the spectrum analysis of those models." The eigenvalue spectra of OPT activations decay faster, meaning PCA concentrates variance into fewer dimensions, making dimension deletion less destructive.

Calibration set size sensitivity: Table 5 replicates Table 1 but with a smaller calibration set (128 sequences instead of 1024). The degradation from reducing calibration data is systematic but modest. For LLAMA-2 70B at 25% slicing: calibration size 1024 achieves 4.60, calibration size 128 achieves 4.89 — a 0.29 increase. This confirms that more calibration data helps but the method is not catastrophically sensitive to calibration set size above ~128 sequences, consistent with the ablation in Appendix A.3 (Figure 7).

Numerical precision sensitivity: Table 4 (Appendix A.2) compares FP32 vs. FP64 PCA computation. The impact is substantial for large models: LLAMA-2 70B at 25% slicing with FP32 PCA achieves 7.01 perplexity vs. 4.60 with FP64 PCA — more than 2.4 points worse. At 30% slicing, the gap widens to 8.75 (FP32) vs. 5.05 (FP64). Smaller models show similar relative patterns: OPT 66B at 25% goes from 9.68 (FP64) to 9.81 (FP32), a smaller but still measurable degradation. The paper notes that "the accuracy of larger models could be affected by numerical errors during the PCA calculation phase" — this is practically important because it means the compression quality ceiling for the largest models is partly determined by linear algebra precision, not just by the spectral properties of the activations.

Zero-Shot Tasks: Downstream Accuracy

Figure 5 (top row: WikiText-2 calibration; bottom row: Alpaca calibration) and Tables 7-10 (Appendices A.5) report accuracy on five zero-shot benchmarks.

Headline numbers for OPT 66B: With WikiText-2 calibration (Table 7), the dense model achieves 66.16% average accuracy. At 20% slicing: 65.74% (99.4% retention). At 25% slicing: 65.17% (98.5% retention). At 30% slicing: 64.24% (97.1% retention). The per-task breakdown shows remarkable stability: PIQA drops from 79.82 to 77.42 (30% slicing), WinoGrande from 68.90 to 66.30, HellaSwag from 74.85 to 72.62, ARC-e from 67.21 to 66.90, ARC-c from 40.02 to 37.97. No single task collapses — the compression evenly affects all capabilities. With Alpaca calibration (Table 8), performance is slightly better: 30% sliced OPT 66B achieves 65.00% vs. 64.24% with WikiText-2 calibration. The paper's abstract claim that SliceGPT "maintains 99% zero-shot task performance" for OPT 66B refers to the 25% slicing result with Alpaca calibration (65.30% vs. 66.16% dense, approximately 98.7% retention).

Headline numbers for LLAMA-2 70B: With WikiText-2 calibration (Table 7): dense 76.57%, 20% sliced 72.34% (94.5% retention), 25% sliced 69.75% (91.1% retention), 30% sliced 66.11% (86.3% retention). With Alpaca calibration (Table 8): dense 76.57%, 20% sliced 74.43% (97.2%), 25% sliced 73.59% (96.1%), 30% sliced 71.67% (93.6%). The abstract's claim of "99% zero-shot task performance" for LLAMA-2 70B at 25% slicing refers to the recovery fine-tuning results (Figure 6, right panel), discussed below — not the zero-shot slicing numbers.

Headline numbers for Phi-2: With WikiText-2 calibration (Table 7): dense 72.24%, 20% sliced 58.15% (80.5%), 25% sliced 54.46% (75.4%), 30% sliced 51.99% (72.0%). Phi-2 degrades much more severely than the larger models, consistent with the paper's observation that smaller models are harder to compress. With Alpaca calibration (Table 8), the pattern is similar: 25% sliced retains 62.52% (86.5% of dense). The abstract's mention of "90% zero-shot task performance" for Phi-2 refers to the 25% slicing with recovery fine-tuning on Alpaca (Table 10): 65.24% vs. 72.24% dense, approximately 90.3% retention.

Cross-task variation: Across all models and slicing percentages, no single task is disproportionately affected. PIQA and WinoGrande tend to be the most robust; HellaSwag shows the largest absolute drops. For LLAMA-2 70B at 25% slicing (WikiText-2): PIQA drops from 82.70 to 74.92 (-7.78), HellaSwag from 83.84 to 68.84 (-15.00). This pattern is consistent with HellaSwag requiring more fine-grained language understanding that may be distributed across more embedding dimensions.

Calibration dataset transfer: Comparing Figure 5 top (WikiText-2 calibration) vs. bottom (Alpaca calibration) reveals that the calibration dataset choice matters for downstream tasks. For LLAMA-2 7B at 25% slicing: WikiText-2 calibration yields 55.48% average accuracy, Alpaca calibration yields 60.91% — a 5.43 percentage point improvement from switching calibration sets. The same pattern holds across model families and sizes. The paper attributes this to distributional alignment: Alpaca's instruction-following format more closely matches the zero-shot benchmarks than WikiText-2's raw text, so PCA computed on Alpaca activations preserves dimensions that matter more for downstream task performance.

Recovery Fine-Tuning (RFT) Results

Figure 6 and Tables 9-10 assess whether LoRA fine-tuning after slicing can recover lost accuracy.

WikiText-2 RFT (Table 9, Figure 6 left): For LLAMA-2 models, RFT on WikiText-2 provides minimal benefit and sometimes hurts. LLAMA-2 70B at 25% slicing: without RFT 69.75%, with RFT 71.28% — a modest 1.53-point gain. LLAMA-2 7B at 25% slicing: without RFT 55.48%, with RFT 56.20% — only 0.72-point gain. The paper notes: "we see a marked difference between RFT on WikiText-2 and Alpaca datasets, with the Alpaca dataset giving much higher performing models."

Alpaca RFT (Table 10, Figure 6 right): Recovery fine-tuning on Alpaca produces substantially larger gains. LLAMA-2 70B at 25% slicing: without RFT 73.59%, with RFT 75.62% — approaching the dense 76.57% (98.8% retention). At 30% slicing with RFT: 74.30% vs. dense 76.57% (97.0% retention). For LLAMA-2 13B at 25% slicing with RFT: 68.18% vs. dense 71.76% (95.0%). For LLAMA-2 7B at 25% slicing with RFT: 63.04% vs. dense 69.00% (91.4%). For Phi-2 at 25% slicing with RFT: 65.24% vs. dense 72.24% (90.3%). The paper highlights Phi-2's result: "the sliced model has approximately 2.2B parameters and retains 90.3% of the accuracy of the 2.8B model. This shows that even small LMs can benefit from post-training pruning" — though the qualification is that RFT is essential for this level of recovery in smaller models.

OPT RFT failure: The paper reports a notable negative result: "Despite an extensive search, we were not able to find RFT parameters that enabled improved performance in the OPT models" (Figure 6 caption). No RFT results for OPT models are reported, implying that LoRA fine-tuning did not reliably improve upon the already-strong zero-shot slicing results for OPT. This asymmetry between OPT and LLAMA-2 under RFT is unexplained in the paper but suggests differences in how the two model families respond to post-compression fine-tuning — possibly related to their different spectral properties or training methodologies.

Inference Speedup and Throughput

Table 2 reports per-token inference time for the largest models at 25% slicing, comparing dense and sliced deployment on two GPU tiers.

Consumer GPU results (Quadro RTX6000, 24GB): For LLAMA-2 70B, the dense model requires 7 GPUs and 252ms per token, consuming 1764 GPUms. After 25% slicing, the model fits on 5 GPUs at 215ms per token, consuming 1075 GPUms — a reduction to 60.9% of the dense model's total compute. For OPT 66B, dense requires 6 GPUs at 237ms per token (1422 GPUms); sliced requires 5 GPUs at 204ms (1020 GPUms), reducing total compute to 71.7%. In both cases, the sliced model requires 2 fewer GPUs, directly lowering the hardware barrier for deployment.

Datacenter GPU results (A100, 40GB): For LLAMA-2 70B, dense requires 4 GPUs at 125ms per token (500 GPUms); sliced requires 3 GPUs at 110ms (330 GPUms) — a reduction to 66% of dense compute. For OPT 66B, dense requires 4 GPUs at 114ms (456 GPUms); sliced requires 3 GPUs at 102ms (306 GPUms) — reduction to 67.1%. The paper's abstract claims — "on 40GB A100 GPUs we reduce it to 66%" for LLAMA-2 70B and "on 24GB consumer GPUs we reduce the total compute for inference on LLAMA-2 70B to 64% of that of the dense model" — align with Table 2, though the consumer GPU figure (64%) is computed as 1075/1764 = 60.9%, suggesting the "64%" in the abstract may derive from a slightly different measurement or rounding.

Throughput benchmarking (Table 11, Appendix A.6): On 80GB H100 GPUs with sequence length 128 and maximum batch size, 25% sliced LLAMA-2 70B achieves 839 tokens/s vs. 541 tokens/s for dense — a 1.55× speedup at the same GPU count (2 GPUs). Critically, the sliced model can double its batch size (from 128 to 256) because reduced activation memory frees GPU RAM. At 50% slicing (where perplexity degrades substantially — Table 1 shows 8.86 for LLAMA-2 70B), the model fits on a single H100 with 1014 tokens/s vs. dense on 2 GPUs at 541 tokens/s — a 3.75× throughput improvement for the same hardware, or 1.87× per GPU. For OPT 66B at 50% slicing: the model drops from 2 GPUs to 1 with throughput increasing from 141 to 441 tokens/s — a 6.26× effective speedup relative to the dense model on the same number of GPUs.

Comparison to SparseGPT 2:4 speedup (Tables 12-13, Appendix A.7): The paper benchmarks individual matrix multiplications in transformer layers to compare SliceGPT speedup against SparseGPT 2:4 at a fine-grained level. For LLAMA-2 70B: dense layer operations total 16.13ms, SparseGPT 2:4 totals 12.20ms (1.32× speedup), SliceGPT 25% totals 12.20ms (1.32× speedup) — identical speedup at 25% slicing but with better perplexity (4.60 vs. 4.98). For LLAMA-2 7B: dense 3.99ms, SparseGPT 2:4 2.70ms (1.48×), SliceGPT 25% 2.99ms (1.33×) — SparseGPT is faster but with worse perplexity (8.69 vs. 7.24). For OPT 66B: dense 14.01ms, SparseGPT 2:4 10.81ms (1.30×), SliceGPT 25% 10.56ms (1.33×) — SliceGPT achieves slightly better speedup and better perplexity (9.68 vs. 10.22). This per-operation analysis reveals that SliceGPT's speedup advantage over SparseGPT grows with model size — for the largest models, SliceGPT matches or exceeds SparseGPT's 2:4 speedup while maintaining lower perplexity, and simultaneously reduces GPU count (which SparseGPT 2:4 cannot do since activation memory is unchanged).

Compute Cost of Compression

Table 3 reports the wall-clock time and GPU requirements for the compression pipeline. For LLAMA-2 70B: PCA computation takes 3 hours 31 minutes on 1× H100 80GB, and recovery fine-tuning takes an additional 1 hour 35 minutes on 4× H100 80GB, totaling approximately 5 hours. For LLAMA-2 7B: 44 minutes for slicing + 23 minutes for RFT = 1 hour 7 minutes total on 1× H100. For Phi-2: 49 minutes for slicing on 1× V100 32GB + 1 hour 59 minutes RFT = approximately 2 hours 48 minutes. The compression is "performed on a single H100 GPU with 80GB of memory" for the PCA step across all models — for LLAMA-2 70B, the full model fits on a single H100 during PCA computation (since only forward passes and eigenvalue calculations are needed, not optimizer states or gradients). This distinguishes SliceGPT from methods requiring multi-GPU training infrastructure; a single datacenter GPU can compress a 70B model in an afternoon.

Ablation Studies and Robustness Checks

Calibration set size and sequence length (Appendix A.3, Figure 7, Table 5): The ablation on calibration configuration uses OPT 6.7B and LLAMA-2 7B at 25% slicing. Increasing calibration set size from 32 to 128 samples sharply reduces perplexity; beyond 128, gains are marginal (Figure 7, left). For sequence length (Figure 7, right), longer sequences consistently improve perplexity up to the tested maximum of 4096. The authors conclude that "having a larger sequence length can result in better perplexity" and adopt 1024 samples × 2048 sequence length for main experiments. Table 5 confirms that reducing calibration size to 128 sequences degrades performance across all models: LLAMA-2 70B 25% slicing goes from 4.60 to 4.89; OPT 66B from 9.68 to 9.76. The degradation is systematic but moderate — the method is not brittle to calibration data quantity.

FP32 vs. FP64 PCA computation (Appendix A.2, Table 4): Using single-precision (FP32) rather than double-precision (FP64) for eigenvector calculation produces substantially worse results for larger models. For LLAMA-2 70B at 25% slicing: FP64 achieves 4.60 perplexity, FP32 achieves 7.01 — a 52% increase in perplexity. At 30% slicing, the gap widens: 5.05 (FP64) vs. 8.75 (FP32). For OPT 66B, the effect is smaller but still present: 25% slicing goes from 9.68 (FP64) to 9.81 (FP32). The paper notes this is due to "numerical errors during the PCA calculation phase" — the eigenvector computation in high dimensions becomes ill-conditioned, and FP32's lower precision propagates errors that compound when the eigenvectors are used as rotation matrices. This is a practically significant finding: for the largest models, FP64 is not optional but required for tolerable results.

Variable vs. constant slicing by layer (Appendix A.4, Table 6): Instead of removing a fixed percentage of dimensions from every layer, the variable-slicing approach sets a threshold on the fraction of total variance to discard during PCA and lets each layer's spectral decay determine how many dimensions are removed. The results are asymmetric across model families. For OPT models, variable slicing improves perplexity: OPT 6.7B at ~25% total slicing goes from 12.10 (constant) to 11.94 (variable); OPT 13B from 11.04 to 10.76; OPT 30B from 10.13 to 9.95; OPT 66B from 9.75 to 9.63. For LLAMA-2 models, variable slicing degrades perplexity: LLAMA-2 7B goes from 6.84 to 7.63; 13B from 6.00 to 6.17; 70B from 4.44 to 4.63. The paper attributes this to differences in spectral structure (Figure 8): OPT models have steeper spectral decay in early layers (making them clearly more compressible), while LLAMA-2 models have flatter, more uniform spectra (so adaptive thresholds fail to find layers that are distinctly more or less compressible). This negative result is informative — it demonstrates that variable slicing is not universally beneficial and that the choice between uniform and adaptive allocation depends on the model family's spectral properties.

Calibration dataset transfer for zero-shot tasks (Tables 7-8, Figure 5): Comparing WikiText-2 vs. Alpaca calibration reveals a consistent pattern: Alpaca calibration yields better downstream accuracy across all model families and slicing percentages. For LLAMA-2 13B at 25% slicing: WikiText-2 calibration 58.90% mean accuracy, Alpaca calibration 65.44% — a 6.54-point improvement. For OPT 66B at 25% slicing: WikiText-2 65.17%, Alpaca 65.30% — a smaller 0.13-point improvement. For Phi-2 at 25%: WikiText-2 54.46%, Alpaca 62.52% — an 8.06-point improvement. The paper explains this as a distribution-shift effect: "We attribute this difference to the similarity between Alpaca and the benchmark tasks" (Section 4.1). PCA on Alpaca activations preserves dimensions that carry instruction-following and reasoning signals relevant to PIQA, WinoGrande, HellaSwag, and ARC; WikiText-2 activations are more heavily weighted toward next-token prediction on encyclopedic text, and dimensions important for that objective need not align with benchmark performance.

Recovery fine-tuning dataset interaction (Figure 6, Tables 9-10): RFT with Alpaca yields substantially larger gains than RFT with WikiText-2. For LLAMA-2 70B at 30% slicing: RFT on WikiText-2 achieves 69.02% (Table 9), RFT on Alpaca achieves 74.30% (Table 10) — a 5.28-point difference. For Phi-2 at 25% slicing: WikiText-2 RFT 55.17%, Alpaca RFT 65.24% — a 10.07-point difference. This interaction between calibration data and RFT data is a practically important finding: to maximize zero-shot task performance, both the PCA calibration set and the recovery fine-tuning data should be drawn from distributions similar to the target tasks. The paper's best results (Figure 6, right) use Alpaca for both.

Dense throughput comparison for 50% slicing (Appendix A.6, Table 11): While 50% slicing degrades perplexity beyond what most applications would tolerate (LLAMA-2 70B: 8.86, up from 3.32; Table 12), it demonstrates the upper bound of SliceGPT's speedup capability. The key finding is that the largest models drop from requiring 2 GPUs to 1 GPU at 50% slicing, with throughput increasing more than the embedding dimension reduction alone would predict — because fitting on a single GPU eliminates inter-GPU communication overhead entirely. For OPT 66B at 50% slicing: 1 GPU achieves 441 tokens/s, while dense required 2 GPUs for 141 tokens/s, yielding $6.26\times$ effective throughput improvement when normalizing by GPU count. The paper does not advocate 50% slicing for practical use but presents it to demonstrate the memory and throughput scaling at aggressive compression ratios.

Per-operation timing comparison with SparseGPT (Appendix A.7, Tables 12-13): This ablation decomposes the speedup into individual matrix multiplications, comparing dense, SparseGPT 2:4, SliceGPT 25%, and SliceGPT 50% for all LLAMA-2 and OPT model sizes. For LLAMA-2 7B, SparseGPT 2:4 achieves 1.48× speedup vs. SliceGPT 25% at 1.33× — SparseGPT is faster for small models. For LLAMA-2 70B, both achieve 1.32× speedup — they are tied. For large OPT models, SliceGPT edges ahead: OPT 66B shows SliceGPT 25% at 1.33× vs. SparseGPT 2:4 at 1.30×. The crossover at larger model sizes is explained by the relative cost of different operations: in small models, the residual connection matrices $Q_{\ell-1}^\top Q_\ell$ (which SliceGPT adds) represent a larger fraction of total compute; in large models, the weight matrix multiplications dominate, and SliceGPT's dimensional reduction on both weights and activations compounds in a way that fixed 2:4 sparsity cannot match.

LayerNorm-to-RMSNorm conversion correctness: The paper does not present an explicit ablation verifying that the LayerNorm-to-RMSNorm conversion is lossless, but this is implicitly validated by the 10% slicing results in Table 1 — at only 10% dimension reduction, the degradation is minimal across all models (OPT 66B: 9.33 → 9.43; LLAMA-2 70B: 3.32 → 3.69), indicating that the transformations prior to slicing are not themselves introducing error. If the LayerNorm absorption introduced numerical approximation error, it would appear at all slicing levels, including 0% and 10%.

Critical Assessment

The experiments provide substantial empirical support for SliceGPT's core value proposition — that orthogonal transformations plus PCA-guided dimension deletion can compress large transformers while maintaining competitive accuracy and achieving real speedup — but several important limitations qualify the strength of the evidence and the breadth of the claims.

Claim: SliceGPT maintains 99% zero-shot accuracy for LLAMA-2 70B and OPT 66B at 25% removal.

This claim holds only after recovery fine-tuning for LLAMA-2 70B, not in the zero-shot setting. The abstract states: "SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA-2 70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively." The data supporting LLAMA-2 70B at 99% retention appears in Table 10: with Alpaca calibration and Alpaca RFT, 25% sliced LLAMA-2 70B achieves 75.62% vs. 76.57% dense — approximately 98.8% retention. Without RFT (Table 8), the same configuration achieves 73.59% — only 96.1% retention. The Phi-2 claim of 90% retention likewise requires RFT (Table 10: 65.24% with RFT vs. 72.24% dense). The abstract's phrasing "maintaining ... zero-shot task performance" could be read as implying zero-shot compression (no fine-tuning), but the numbers cited require RFT. This is a meaningful distinction because RFT adds 1.5 hours and 4 GPUs for LLAMA-2 70B (Table 3), changing the cost profile of compression.

The OPT 66B claim of 99% retention is supported without RFT: Table 7 shows 25% sliced OPT 66B at 65.17% vs. 66.16% dense (98.5%), and Table 8 shows 65.30% vs. 66.16% (98.7%). For OPT, RFT was attempted but "despite an extensive search, we were not able to find RFT parameters that enabled improved performance" — making the zero-shot results the best available. This asymmetry between OPT and LLAMA-2 is underexplored: why does RFT help LLAMA-2 but not OPT? The spectral differences (Figure 8) offer a partial explanation — OPT's PCs retain more information, so there's less lost accuracy to recover — but this is not directly tested.

Claim: SliceGPT reduces LLAMA-2 70B inference compute to 64% (consumer) and 66% (A100) of dense.

These numbers are well-supported by Table 2. However, the paper acknowledges an important caveat in a footnote: "Our Hugging Face-based testing does not enjoy continuous batching or model sharding. This means that in terms of inference time, the dense-model could be improved more than our sliced model in terms of GPUms." The speedup measurements use a straightforward Hugging Face + PyTorch implementation without production-grade optimizations. The dense model — being a standard, unmodified architecture — may benefit more from tensor parallelism, pipeline parallelism, and kernel fusion than the sliced model, whose residual connection matrices $Q_{\ell-1}^\top Q_\ell$ introduce an atypical operation that may not be as optimized in inference frameworks. The paper argues that the GPUms metric "does reflect the energy-usage per token in such a deployment," which is reasonable, but the practical speedup experienced by a user of an optimized serving system could be smaller than Table 2 suggests.

The batch-size-1 latency measurements (Table 2) also do not reflect the throughput-oriented regime where SliceGPT's memory reduction provides the largest gains via batch-size scaling. Table 11 partially addresses this by showing maximum-throughput benchmarks, but only for H100 GPUs and without comparison to optimized sparse kernels for SparseGPT. The paper's core speedup claim is that SliceGPT reduces both the number of GPUs required and the per-token latency — the GPU count reduction is unambiguous and valuable, but the per-token latency improvement on optimized inference stacks remains incompletely characterized.

Claim: SliceGPT outperforms SparseGPT 2:4 in perplexity at equivalent or better speedup.

This holds for the largest models (LLAMA-2 70B, OPT 66B) where SliceGPT matches or slightly exceeds SparseGPT 2:4's speedup while achieving lower perplexity. For smaller models (LLAMA-2 7B, OPT 13B), SparseGPT 2:4 is faster (Table 12: 1.48× vs. 1.33× for LLAMA-2 7B) but SliceGPT maintains lower perplexity. The tradeoff is a spectrum: at small scales, SliceGPT favors accuracy over speedup relative to SparseGPT; at large scales, SliceGPT wins or ties on both axes. The paper acknowledges this implicitly: "our pruned models have more parameters than those pruned with SparseGPT but our method allows for larger batch sizes to be loaded into GPU memory" (Section 5). The parameter-count comparison needs clarification: SparseGPT 2:4 keeps all matrix dimensions at $D \times D$ but zeros 50% of entries; SliceGPT 25% reduces dimensions to $0.75D$ but keeps all remaining entries non-zero. For a $D \times D$ matrix, SparseGPT 2:4 stores $0.5 D^2$ non-zero values (plus sparsity metadata), while SliceGPT stores $(0.75)^2 D^2 = 0.5625 D^2$ values as a dense matrix. SliceGPT actually has more stored parameters for the smaller weight matrices, but gains memory from reduced activations and gains speed from dense kernel efficiency. This nuance — that SliceGPT is not strictly sparser in parameter count — means the "compression" is better understood as dimensionality reduction that enables memory and compute savings through smaller tensors, not through fewer stored values.

Missing experiments and baselines:

The paper does not compare against structured pruning methods with recovery fine-tuning like LLM-Pruner (Ma et al., 2023a) or LLM Surgeon (van der Ouderaa et al., 2023), which are the closest methodological competitors. Both are acknowledged in the related work (Section 2.2) but no head-to-head comparison is provided. A comparison against a same-budget structured pruning method would contextualize whether SliceGPT's 3.5-hour one-shot compression is more efficient than interleaved prune-and-fine-tune approaches.

The evaluation is limited to a single generation benchmark (WikiText-2) and five zero-shot tasks. Missing are evaluations on: (a) few-shot tasks that probe in-context learning capability; (b) generation quality metrics (e.g., MT-Bench, AlpacaEval) that would assess whether slicing degrades open-ended response quality; (c) reasoning benchmarks (e.g., GSM8K, MMLU) that test whether the compressed model retains complex reasoning abilities. The zero-shot tasks used (PIQA, WinoGrande, HellaSwag, ARC) are standard in the compression literature but are primarily commonsense reasoning benchmarks that may not stress the full capabilities of a 70B model. A model could lose substantial reasoning ability while maintaining strong performance on these specific benchmarks, and the current evaluation would not detect this.

The calibration-to-evaluation distribution shift is explored only through the WikiText-2 vs. Alpaca comparison, but the implications are not fully characterized. The paper shows that Alpaca calibration improves downstream task performance, suggesting that the PCA basis is data-dependent. This means a practitioner using SliceGPT must select a calibration dataset representative of their deployment distribution — but the paper does not provide guidance on how to choose calibration data, how much is needed, or what happens under distribution shift between calibration and deployment. A robustness study evaluating sliced models on out-of-distribution text (e.g., domain-specific corpora, code, non-English text) would substantially strengthen the practical deployment case.

The single-GPU PCA computation claim (3.5 hours for LLAMA-2 70B) is for FP64 precision on an 80GB H100. The FP32 vs. FP64 ablation (Table 4) shows FP32 is substantially worse for large models, meaning FP64 is required. But FP64 matrix operations are slower and consume more memory than FP32 — the paper does not report whether FP64 PCA on a 70B model fits on GPUs with less than 80GB, which would limit the method's accessibility for researchers without top-tier hardware despite the paper's emphasis on consumer-GPU deployment of the compressed model.

The residual connection overhead is never isolated for ablation. The per-block Q matrices require inserting $Q_{\ell-1}^\top Q_\ell$ into the residual path, which the paper acknowledges adds "a small ($D \times D$) overhead" (Section 3.3). But the magnitude of this overhead relative to the weight matrix savings is never quantified. For a 25% sliced model, the original residual connection was a no-op (identity mapping); the transformed residual connection requires a $0.75D \times 0.75D$ matrix multiplication at every layer. How much of the inference time is spent in these operations versus the FFN and attention computations? If a large fraction, further optimization (e.g., fusing the residual transformation with the preceding LayerNorm) could yield additional gains. The paper does not report this breakdown.

Statistical robustness: All results are single-run evaluations. The test set is the standard WikiText-2 test set (for perplexity) and the standard LM Evaluation Harness splits (for zero-shot tasks), so the evaluation data is fixed and established. However, the PCA computation involves randomness from the calibration set sampling (which sequences are selected) and potentially from numerical variation in eigendecomposition. The paper does not report whether results are stable across different calibration set draws or different random seeds, which matters for reproducibility — a practitioner running SliceGPT on their own model might get different perplexity due to calibration set sampling variation.

The variable-slicing negative result for LLAMA-2 (Table 6) is informative but underexplored. The paper attributes it to flatter spectra but does not test whether alternative adaptive strategies (e.g., slicing based on eigenvalue gaps rather than variance thresholds, or using block-level held-out perplexity as a selection criterion) would work better for LLAMA-2 models. This is a lost opportunity to provide actionable guidance for practitioners working with LLAMA-2 family models.

Scale extrapolation: The largest tested model is 70B parameters. The paper's finding that larger models compress better is consistent and monotonic but is based on three data points for LLAMA-2 (7B, 13B, 70B) and an incomplete OPT series (up to 66B, with 175B excluded). Whether the trend continues to models with 175B+, 405B+, or mixture-of-expert architectures is untested. The paper "anticipates that this larger model will yield improved results" for OPT 175B, but "anticipation" is not evidence, and the LLAMA-2 70B results showing greater degradation than OPT 66B at equivalent percentages suggest caution in extrapolating across model families.

Overall, the experiments demonstrate that SliceGPT achieves a genuine and practically meaningful tradeoff: substantial reduction in GPU count and per-token latency, with accuracy retention that is competitive with or superior to SparseGPT 2:4 at large model scales, and with the additional benefit of activation memory reduction that enables larger batch sizes. The evidence is strongest for the largest models (66B–70B parameters) on the WikiText-2 generation task. The evidence is weaker for smaller models, for downstream task performance without RFT (particularly LLAMA-2 and Phi-2), and for generalization to tasks and deployment conditions not explicitly tested. The experiments successfully validate the core mechanism — PCA-guided orthogonal transformation enables principled embedding dimension reduction — but leave open the question of whether SliceGPT's compression approach is robust to the full diversity of LLM use cases beyond the five zero-shot benchmarks evaluated.

6. Limitations and Trade-offs

6.1 The Calibration Dataset Defines the Compression Basis — and There Is No Way to Know It Is Right Without Evaluating

The assumption or constraint. SliceGPT computes the orthogonal transformation $Q_\ell$ at each block via PCA on activations collected from a calibration dataset. The choice of calibration data determines which dimensions are preserved and which are sliced away. The paper uses WikiText-2 and Alpaca as calibration sets and observes that the choice matters: Alpaca-calibrated models perform substantially better on zero-shot downstream tasks than WikiText-2-calibrated models (Figure 5, Tables 7–8). For LLAMA-2 7B at 25% slicing, switching calibration from WikiText-2 to Alpaca improves mean accuracy from 55.48% to 60.91% — a 5.43 percentage point gain. For Phi-2 at the same slicing level, the improvement is 54.46% to 62.52% — over 8 points.

The paper explains this as a distribution-shift effect: "We attribute this difference to the similarity between Alpaca and the benchmark tasks" (Section 4.1). But this explanation reveals the assumption rather than resolving it: the compressed model's quality depends on the alignment between calibration data and deployment data. The paper does not provide a principled method for selecting calibration data, nor does it characterize how sensitive the method is to calibration-deployment mismatch beyond the two datasets tested. A practitioner who calibrates on WikiText-2 but deploys on code generation, or medical text, or multilingual data, has no guidance on what accuracy degradation to expect.

The consequence. The practical failure mode is clear: if the calibration distribution differs systematically from the deployment distribution, the PCA basis will preserve dimensions that are irrelevant to the target task and discard dimensions that matter. This is not a small effect — the Alpaca vs. WikiText-2 gap of 5–8 accuracy points on zero-shot tasks is comparable to the degradation from 25% slicing itself. A practitioner who chooses the wrong calibration set could experience substantially worse performance than the paper's headline results, with no diagnostic signal until evaluation on the deployment task is performed.

More subtly, even with well-matched calibration data, the PCA criterion optimizes for reconstruction of the activation vectors, not for task performance. A dimension could carry little variance in activation space (and thus be sliced) but encode information critical for a specific downstream capability — e.g., a rare token type, a syntactic structure, or a factual association that appears infrequently in the calibration data. The paper's zero-shot evaluation partially addresses this concern by showing that average accuracy is maintained, but it does not test whether specific types of capabilities are disproportionately affected — a model could lose the ability to handle long-range dependencies, numerical reasoning, or low-resource languages while maintaining aggregate perplexity and commonsense QA scores.

What evidence exists in the paper. The Alpaca vs. WikiText-2 calibration comparison (Figure 5, Tables 7–8) provides direct evidence of calibration dependence. The recovery fine-tuning interaction (Figure 6) is also informative: RFT on Alpaca helps substantially more than RFT on WikiText-2, suggesting that even after fine-tuning, the initial PCA basis (and thus the calibration choice) constrains what can be recovered. The spectrum analysis (Appendix A.4, Figure 8) shows that eigenvalue distributions differ by calibration data, but this is presented as an observation rather than as a diagnostic tool for calibration selection.

Mitigation status. The paper does not attempt to address calibration set selection. It tests two datasets and reports results for both, which is transparent, but it does not propose a method for choosing calibration data, for estimating the sensitivity of a given model-data pair, or for making the PCA basis robust to distribution shift. The recovery fine-tuning results (Figure 6, Tables 9–10) partially mitigate the issue — RFT on task-relevant data can recover some of the accuracy lost from poor calibration choices — but RFT cannot fully compensate for dimensions that were already sliced and whose information is permanently lost. The paper also suggests, in Section 5, that "other methods of computing $Q$ could improve the results," which implicitly acknowledges that PCA on a fixed calibration set may not be optimal, but no alternatives are developed.

6.2 Smaller Models and Models with Flat Spectra Are Poor Candidates for SliceGPT

The assumption or constraint. SliceGPT's compression mechanism depends on the eigenvalue spectrum of the activation covariance matrix $C_\ell$ decaying sufficiently rapidly that discarding the trailing dimensions loses negligible information. The paper's own analysis (Appendix A.4, Figure 8) shows that this spectral property varies substantially across model families and across model sizes. LLAMA-2 models have flatter spectra than OPT models at comparable parameter counts; smaller models have flatter spectra than larger models within the same family; and later layers have flatter spectra than early layers.

The consequence of a flat spectrum is that PCA cannot concentrate information into a small number of leading dimensions — variance is genuinely distributed across all $D$ dimensions. Slicing any fixed percentage therefore deletes a proportionally large amount of information, and perplexity degrades rapidly. The paper's results confirm this pattern:

• Smaller models degrade more: OPT 125M at 25% slicing goes from 27.64 to 37.74 perplexity (+10.10), while OPT 66B at the same slicing goes from 9.33 to 9.68 (+0.35). The absolute degradation in perplexity points is 29× larger for the 125M model despite only ~0.2% the parameter count. LLAMA-2 follows the same trend: 7B degrades by 1.77 perplexity points, 70B by 1.28 points, and the relative degradation (percentage increase) is 32% vs. 39% — a less dramatic difference but still favoring larger models.

• LLAMA-2 degrades more than OPT: At 25% slicing, OPT 66B loses 3.7% of its dense perplexity; LLAMA-2 70B loses 38.6%. The paper attributes this directly to spectral differences: "SliceGPT exhibits superior performance when applied to OPT models compared to LLAMA-2 models which matches our intuition from the spectrum analysis of those models" (Section 4.1).

• Variable slicing hurts LLAMA-2: The attempt to adapt slicing percentages per layer (Table 6) improves OPT models but degrades LLAMA-2 models, consistent with LLAMA-2's flatter, more uniform spectra where no layers are clearly more compressible than others.

The consequence. SliceGPT is not a universal compression method suitable for all models. It is specifically effective for large models with rapid spectral decay — which, practically, means the largest OPT variants and, to a lesser extent, LLAMA-2 70B with recovery fine-tuning. For models below approximately 7B parameters, or for model families with inherently distributed representations, SliceGPT at compression ratios that provide meaningful speedup (25%+) may cause unacceptable accuracy degradation. A practitioner considering SliceGPT for a 7B model should expect, based on Table 7, that 25% slicing will reduce zero-shot accuracy from 69.00% to 55.48% (LLAMA-2 7B) without RFT — a loss of ~13.5 percentage points that is unlikely to be acceptable in most deployment scenarios.

The Phi-2 results (2.8B parameters) further emphasize this limitation. At 25% slicing without RFT, Phi-2 drops from 72.24% to 54.46% — a loss of nearly 18 percentage points. Even with Alpaca RFT (Table 10), the recovery only reaches 65.24% (90.3% of dense). The paper presents this as a positive: "this shows that even small LMs can benefit from post-training pruning" (Section 4.1). But a practitioner might reasonably view 90% accuracy retention with RFT (requiring additional fine-tuning time and compute) as a marginal outcome, especially if the dense 2.8B model is already compact enough for many deployment scenarios.

What evidence exists in the paper. Table 1 provides the systematic scale-dependence evidence across seven OPT sizes and three LLAMA-2 sizes. Figure 5 shows the downstream-task analog. Figure 8 is the key diagnostic exhibit: the normalized eigenvalue spectra visually demonstrate the flatter distributions for LLAMA-2 and for later layers. Table 6 (variable slicing) provides convergent evidence that LLAMA-2 models do not have the spectral structure that makes adaptive compression beneficial.

Mitigation status. The paper acknowledges this limitation implicitly by noting the model-size trend and the spectral analysis, but it does not provide guidance on how to determine before compression whether a given model will compress well. The spectrum analysis in Figure 8 is presented as a post-hoc explanation, not as a pre-compression diagnostic. A practitioner would need to implement their own spectral analysis on calibration data before deciding whether to use SliceGPT. The paper also does not explore whether architectural modifications that induce faster spectral decay (e.g., training with dimensionality-reduction objectives) could improve compressibility. The variable-slicing attempt (Table 6) represents a partial mitigation — adapting the compression ratio to spectral properties — but this approach fails for exactly the model family (LLAMA-2) that needs it most.

6.3 The Per-Block Residual Connection Matrices Add Inference Overhead Not Fully Characterized

The assumption or constraint. Extending the computational invariance to per-block orthogonal matrices requires inserting a linear transformation $Q_{\ell-1}^\top Q_\ell$ into each residual connection (Section 3.3, Figure 4). Unlike the transformations absorbed into the weight matrices — which are pre-computed at compression time and require zero additional runtime computation — these residual connection matrices must be applied during every forward pass. Each one is a $D_{\text{small}} \times D_{\text{small}}$ matrix multiplication at every layer, for every token in the sequence.

The paper acknowledges this overhead: "Unlike the modifications to the weight matrices, these additional operations cannot be pre-computed and add a small ($D \times D$) overhead to the model. Nonetheless, they are needed to allow slicing the model (Section 3.4) and we see real speedup overall (Section 4)." The characterization as "small" is qualitative and never quantified.

The consequence. The residual connection matrices introduce a new category of computation that does not exist in the original transformer architecture. In the original network, a residual connection is a no-cost addition operation: $X \leftarrow X + Z$. In the sliced network, it becomes $X \leftarrow (Q_{\ell-1}^\top Q_\ell) X + Z$, which is a dense matrix-vector multiplication applied to every token at every layer. For a 25% sliced LLAMA-2 70B with $D_{\text{small}} = 0.75 \times 8192 = 6144$, each residual matrix is $6144 \times 6144$, and there are 79 such matrices (one per layer, since LLAMA-2 70B has 80 layers and the last layer feeds into the head). For a sequence of 2048 tokens, the residual transformations add $79 \times 2048 \times 6144^2 \approx 6.1 \times 10^{12}$ FLOPs per forward pass — a non-trivial addition.

The practical concern is that this overhead disproportionately affects certain deployment scenarios:

• Latency-sensitive applications (batch size 1): The residual transformations add latency to every token generation step that cannot be parallelized across tokens during autoregressive decoding. Table 2's batch-size-1 measurements include this overhead, and the speedup (16–17% on consumer GPUs) is lower than the theoretical FLOP reduction from smaller weight matrices would suggest, likely due in part to the residual overhead.
• Throughput-oriented applications (large batch size): Here, the weight matrix multiplications (which benefit from the embedding dimension reduction) dominate, and the residual overhead is a smaller fraction of total compute. Table 11 shows larger speedups for throughput (1.55× at 25% slicing for LLAMA-2 70B) than Table 2 shows for latency, consistent with this analysis.
• The residual matrices create an irregular computation pattern: Unlike the regular sequence of attention and FFN blocks, the residual transformations are a new operation that may not benefit from existing kernel fusion optimizations in inference frameworks. The paper's footnote acknowledges this: "Our Hugging Face-based testing does not enjoy continuous batching or model sharding. This means that in terms of inference time, the dense-model could be improved more than our sliced model in terms of GPUms."

What evidence exists in the paper. No explicit measurement. The paper does not report what fraction of inference time is spent in the residual transformations versus the weight matrix multiplications. The per-operation timing in Appendix A.7 (Tables 12–13) benchmarks individual weight matrix multiplications (down projection, up/gate projection, K/V, Q, output) but does not include the residual connection operations. This omission makes it impossible to determine from the paper whether the residual overhead is 2%, 10%, or 20% of total inference time. The overall speedup numbers in Table 2 and Table 11 suggest the overhead is manageable — real speedup is achieved — but the lack of decomposition prevents understanding of how much more speedup could be obtained if the residual operations were optimized or eliminated.

Mitigation status. Not addressed. The paper does not propose techniques to reduce the residual connection overhead, such as:

• Fusing the residual matrix multiplication with the preceding LayerNorm/RMSNorm operation (since RMSNorm is element-wise and could be combined with the matrix multiplication in a custom kernel).
• Approximating $Q_{\ell-1}^\top Q_\ell$ with a low-rank or structured matrix to reduce the $O(D^2)$ cost.
• Absorbing $Q_{\ell-1}^\top Q_\ell$ into the attention/FFN computations when possible, avoiding a separate kernel launch.

The paper treats the residual matrices as an unavoidable cost of per-block transformations and relies on the overall speedup (which is positive) to argue that the approach is practical. This is a reasonable position for an initial paper establishing the technique, but it leaves a clear optimization opportunity for follow-up work.

6.4 Recovery Fine-Tuning Is Required for Competitive LLAMA-2 and Phi-2 Results, Blurring the "One-Shot" Distinction

The assumption or constraint. The paper positions SliceGPT as a one-shot compression method: "we compress large models using a single GPU in just a few hours and maintain competitive performance on generation and downstream tasks even without RFT" (Section 1). This positioning is central to SliceGPT's value proposition relative to methods like LLM-Pruner and LLM Surgeon, which require interleaved pruning and fine-tuning.

The results tell a more nuanced story. For OPT models, the one-shot claim holds well: Table 7 shows OPT 66B at 25% slicing retains 98.5% of dense accuracy without RFT. For LLAMA-2 and Phi-2, the picture is different:

• LLAMA-2 70B at 25% slicing without RFT: 69.75% vs. 76.57% dense (91.1% retention) — Table 7. With RFT: 75.62% (98.8% retention) — Table 10.
• LLAMA-2 7B at 25% slicing without RFT: 55.48% vs. 69.00% dense (80.4% retention) — Table 7. With RFT: 63.04% (91.4% retention) — Table 10.
• Phi-2 at 25% slicing without RFT: 54.46% vs. 72.24% dense (75.4% retention) — Table 7. With RFT: 65.24% (90.3% retention) — Table 10.

The abstract's headline claims — "99% zero-shot task performance" for LLAMA-2 70B and "90% zero-shot task performance" for Phi-2 — are only achieved after recovery fine-tuning. Without RFT, the numbers are approximately 91% and 75%, respectively — substantially worse and, in Phi-2's case, potentially unacceptable for deployment.

The consequence. A practitioner reading the abstract might reasonably conclude that SliceGPT achieves 99% accuracy retention on LLAMA-2 70B in its one-shot configuration. This is not the case. To reach 99%, the practitioner must run an additional recovery fine-tuning step that takes 1.5 hours on 4× H100 GPUs for LLAMA-2 70B (Table 3). This doubles the total compression time from ~3.5 to ~5 hours and requires multi-GPU infrastructure, partially undermining the "single GPU in just a few hours" narrative. For Phi-2, RFT takes nearly 2 hours on a single V100, more than doubling the compression time.

The distinction matters practically because recovery fine-tuning introduces additional complexity: the practitioner must choose fine-tuning hyperparameters (LoRA rank, learning rate, number of steps), select fine-tuning data, and validate that fine-tuning doesn't introduce regressions on specific tasks. The paper notes that RFT hyperparameter search was non-trivial: "Despite an extensive search, we were not able to find RFT parameters that enabled improved performance in the OPT models" (Figure 6 caption). If RFT hyperparameters are model-family-dependent and require search, the compression pipeline is no longer a simple, automated procedure.

What evidence exists in the paper. Tables 7 and 8 (without RFT) vs. Tables 9 and 10 (with RFT) provide the direct evidence. Figure 6 visualizes the gap. The paper is transparent about the RFT requirement for LLAMA-2 and Phi-2 in the experimental section, but the abstract's language elides the dependency on RFT for the headline numbers. Table 3's compute cost breakdown includes RFT time and GPU count, so the additional cost is documented but is presented separately from the core SliceGPT pipeline.

Mitigation status. Not mitigated. The paper does not investigate why RFT is necessary for LLAMA-2 but not OPT, nor does it propose a variant of SliceGPT that could achieve RFT-level performance without fine-tuning. The spectral analysis (Figure 8) provides a partial explanation — LLAMA-2's flatter spectra mean slicing loses more information that must be recovered through weight adaptation — but this is descriptive, not prescriptive. For a practitioner working with LLAMA-2 or another model family with distributed representations, there is currently no way to achieve the paper's headline accuracy retention figures without the additional RFT step that the paper's one-shot positioning deemphasizes.

6.5 Evaluation Is Limited to a Narrow Slice of LLM Capabilities and Does Not Probe Sensitive Failure Modes

The assumption or constraint. All experimental evaluation uses WikiText-2 perplexity and five zero-shot commonsense reasoning tasks (PIQA, WinoGrande, HellaSwag, ARC-e, ARC-c). These benchmarks are standard in the model compression literature and allow direct comparison with prior work, but they represent a narrow subset of the capabilities that make LLAMA-2 70B and OPT 66B valuable in practice. A model that maintains 99% accuracy on PIQA and ARC may still be substantially degraded on capabilities that these benchmarks do not test.

The consequence. Several practically important capability dimensions are untested:

• Complex reasoning: The paper does not evaluate on GSM8K (grade-school math), MMLU (multi-task knowledge), or any multi-step reasoning benchmark. SliceGPT removes embedding dimensions globally — if certain dimensions encode information critical for mathematical operations, logical deduction, or factual recall, these capabilities could degrade disproportionately while commonsense QA scores remain stable.
• Few-shot / in-context learning: A key capability of large LMs is the ability to adapt to new tasks from a few examples in the prompt. The paper's zero-shot evaluation does not test whether slicing preserves this capability, and the mechanism (per-block PCA on calibration activations) might remove dimensions that are important for recognizing task structure from in-context examples.
• Long-form generation quality: LLMs are often evaluated on the quality of generated text (coherence, factuality, stylistic appropriateness), not just multiple-choice accuracy. The WikiText-2 perplexity metric partially captures language modeling quality, but perplexity improvements do not always correlate with human-judged generation quality. Models compressed via dimensionality reduction might produce more repetitive, less diverse, or less fluent text even if perplexity is well-preserved.
• Code generation: Many deployment scenarios for large LMs involve code generation. The paper does not evaluate on HumanEval, MBPP, or any coding benchmark.
• Multilingual performance: The calibration datasets (WikiText-2, Alpaca) are English-only. PCA on English activations may preserve dimensions that are English-specific and discard dimensions that encode cross-lingual representations, degrading performance on non-English text.

The paper acknowledges the scope limitation implicitly by selecting benchmarks consistent with prior work, but it does not discuss why these benchmarks are sufficient or what risks are undetected.

What evidence exists in the paper. The five-task zero-shot evaluation (Figure 5, Tables 7–10) shows that, within the set of tested capabilities, degradation is roughly uniform — no single task collapses while others remain strong. This provides some reassurance that slicing does not selectively destroy specific reasoning types. But the tasks are all in the same genre (multiple-choice commonsense QA), so the uniformity across them does not imply uniformity across untested capabilities. There is no evaluation that would reveal a selective degradation on, for example, mathematical reasoning or factual knowledge.

Mitigation status. Not addressed. The paper does not suggest that the current evaluation is incomplete or propose extensions to other task types. This is understandable for an initial paper establishing a new compression technique — comprehensive evaluation across dozens of benchmarks is expensive and prior compression work (SparseGPT, GPTQ) similarly uses a focused evaluation suite. However, for a method that physically deletes dimensions from the model's representations, the risk of capricious capability loss is arguably higher than for methods that merely sparsify or quantize weights while preserving the full embedding dimension. A sparse model can, in principle, still route information through any of the original $D$ dimensions; a sliced model has permanently lost those dimensions. A practitioner deploying a sliced model in a high-stakes application would need to evaluate on their specific task distribution, and the paper provides no guidance on what degradation patterns to expect for task types beyond those tested.

6.6 The Difficulty Estimation Overhead Is Not Accounted For in Compression Cost — and Scales Poorly

The assumption or constraint. Computing the PCA basis for each block requires running the full calibration dataset (1024 sequences of length 2048 = ~2M tokens) through the entire model — not just once, but sequentially per block if per-block transformations are applied correctly (since the signal at block $\ell$ depends on the $Q$ matrices applied to all previous blocks). The paper states: "We use the output of the transformed network to calculate the orthogonal matrices of the next layers" (Section 3.3). This means the PCA computation for an $L$-layer model requires approximately $L$ forward passes through the partially-transformed network, each processing the full calibration set.

The paper reports a single-H100 runtime of 3 hours 31 minutes for LLAMA-2 70B (Table 3), but this number is for the end-to-end slicing pipeline and does not decompose how much of that time is calibration forward passes versus eigendecomposition versus weight matrix transformation. Still, it establishes a lower bound: compressing a 70B model takes hours of GPU time before any speedup is realized.

The consequence. The compression cost scales with model size in ways that may limit SliceGPT's applicability to even larger models. Consider a hypothetical 405B-parameter model (approximately 5.8× larger than LLAMA-2 70B by parameter count, with proportionally larger embedding dimension $D$). The PCA computation cost has several components:

• Calibration forward passes: Each forward pass costs $O(L N D^2)$ FLOPs, where $L$ is layers, $N$ is sequence length × number of sequences, and $D$ is embedding dimension. This scales linearly with model parameters (since parameters are $O(L D^2)$) so the forward pass cost scales roughly with parameter count — a 405B model would take roughly 5.8× the forward pass time, or ~20 hours for the calibration passes alone.
• Covariance matrix construction: Computing $C_\ell = \sum_i X_{\ell,i}^\top X_{\ell,i}$ for each layer requires $O(B N D^2)$ FLOPs per layer, where $B$ is batch size and $N$ is sequence length. This also scales with $D^2$ and becomes substantial for large $D$.
• Eigendecomposition: Computing eigenvectors of a $D \times D$ matrix costs $O(D^3)$. For LLAMA-2 70B with $D = 8192$, this is $8192^3 \approx 5.5 \times 10^{11}$ operations per layer, manageable in FP64. For a model with $D = 16384$ (plausible for a 405B-parameter model), the cost increases to $16384^3 \approx 4.4 \times 10^{12}$ operations per layer — 8× more expensive per layer, and there are likely more layers. The FP64 eigendecomposition could become the dominant cost for very large models.

The paper's FP64 requirement (Appendix A.2, Table 4) compounds this scaling concern. FP64 operations are slower and consume more memory than FP32 — an H100's FP64 tensor core throughput is 67 TFLOPS vs. 989 TFLOPS for FP16. If eigendecomposition becomes the bottleneck, the compression time could grow super-linearly with model size, potentially making SliceGPT impractical for the next generation of LLMs.

What evidence exists in the paper. Table 3 provides point estimates for three model sizes, showing that compression time does scale with model size but without enough data points to establish the scaling exponent. Appendix A.2's FP32 vs. FP64 comparison (Table 4) establishes that FP64 is necessary for quality but does not decompose where time is spent. There is no ablation studying how compression time breaks down across calibration, covariance construction, eigendecomposition, and weight absorption.

Mitigation status. Not addressed. The paper does not discuss the scaling properties of the compression algorithm itself, propose approximations to the eigendecomposition (e.g., randomized SVD, power iteration for only the top $D_{\text{small}}$ eigenvectors, or iterative methods that avoid the full $O(D^3)$ cost), or explore whether the sequential per-block computation can be parallelized. The paper's framing — 3.5 hours on a single H100 — presents this as a strength relative to methods requiring distributed training, but for models an order of magnitude larger, the unoptimized PCA pipeline could become a bottleneck that requires its own optimization. This is a forward-looking concern rather than a criticism of the current method, but it is practically relevant given the rapid growth in LLM scale and the fact that SliceGPT's advantages are most pronounced at the largest model sizes.

7. Implications and Future Directions

How This Work Changes the Landscape

SliceGPT introduces a genuinely new axis of model compression — embedding dimension reduction via algebraic invariance — that is categorically distinct from the four canonical approaches of distillation, tensor decomposition, pruning, and quantization. This is not an incremental refinement of existing pruning techniques. It is a recognition that pre-trained transformers possess a previously unexploited degree of freedom: the ability to rotate the entire network's coordinate system without changing any output, enabling the network to be expressed in a basis where dimension deletion is principled rather than catastrophic.

The magnitude of this shift is best understood by what it makes obsolete as a question. Before SliceGPT, the question "can we delete entire rows and columns from a trained transformer's weight matrices?" had a clear empirical answer: no — the paper's own norm-based column pruning baseline produced perplexity in the thousands after deleting just a few columns (Section 4). This negative result was consistent with the intuition that every dimension in a trained network carries distributed, interdependent information. SliceGPT changes the question to: "can we find a rotation where deleting rows and columns is harmless?" — and the answer, proven mathematically in Theorem 1 and demonstrated empirically across model families, is yes. The invariance theorem transforms column deletion from a destructive operation into a principled one by revealing that the network's computation depends only on the subspace spanned by the signal, not on the particular basis used to represent it. This is a reframing of the compression problem from "what can we remove without breaking the function?" to "what coordinate system makes removal harmless?"

The conceptual contribution extends beyond SliceGPT itself. The computational invariance theorem (Theorem 1) establishes that RMSNorm-connected transformers form an equivalence class under orthogonal transformations — any two networks related by $\tilde{W} = f(W, Q)$ for orthogonal $Q$ compute the identical function. This is not an approximation or a heuristic; it is an exact algebraic property of the architecture. The implications of this equivalence class extend beyond compression: it means that a pre-trained model's weight configuration is not a unique solution but one representative of an infinite family, and that different representatives may have different practical properties (compressibility, quantizability, hardware efficiency) while computing exactly the same function. The paper explicitly gestures toward this broader significance: "We hope that our observation of computational invariance can help future research in improving the efficiency of deep learning models, and perhaps inspire new theoretical insights" (Section 5). This is not boilerplate — it identifies a structural property of transformers that was always present but never previously exploited, and the downstream research directions it opens are concrete.

The paper also resolves a latent tension in the compression literature between structured pruning methods that require extensive retraining and one-shot methods that are limited to unstructured or semi-structured sparsity patterns. Prior work on structured pruning for LLMs (LLM-Pruner, LLM Surgeon) demonstrated that removing connected components could achieve good compression ratios, but required interleaved fine-tuning — making them expensive and complex to apply. One-shot methods like SparseGPT achieved reasonable quality without fine-tuning but were restricted to weight-level sparsity patterns that provide limited hardware speedup and do not reduce activation memory. SliceGPT demonstrates that structured compression (dimension reduction) can be achieved in one shot by exploiting a mathematical property of the architecture rather than by compensating for damage through retraining. This changes the cost-benefit calculus for practitioners: structured compression no longer requires the infrastructure and expertise needed for large-scale fine-tuning.

The paper's PCA-guided rotation further shifts the conversation around "saliency criteria" in pruning. Rather than asking which individual weights are important (magnitude pruning, OBS) or which structures have the most impact on the loss (Hessian-based structured pruning), SliceGPT asks: in what rotated basis is the signal most concentrated, and which dimensions of that basis can we discard? The criterion is statistically principled (PCA minimizes L2 reconstruction error) and computationally cheap (no gradients, no Hessian, no retraining). The spectral analysis in Appendix A.4 (Figure 8) demonstrates that this criterion is not just theoretically elegant but empirically predictive: the eigenvalue decay rate directly explains why OPT compresses better than LLAMA-2, why larger models compress better than smaller ones, and why early layers are more compressible than later ones. This provides a diagnostic capability that prior compression work lacked — you can estimate how compressible a model will be before applying compression, simply by examining the activation spectrum on calibration data.

For the broader research landscape, SliceGPT makes certain directions more attractive and others less so. It strengthens the case for methods that exploit architectural invariants — symmetries in the network's computation that can be used to improve efficiency without changing outputs. It weakens the case for purely heuristic pruning criteria (magnitude, random) by demonstrating that a principled, statistically-grounded criterion (PCA) is both more effective and computationally tractable at scale. It also redirects attention from sparsity pattern engineering — which is ultimately constrained by hardware support for specific patterns — toward dimension reduction as a compression strategy that uses standard dense kernels and benefits from all existing hardware optimizations with no specialized kernel requirements. The paper's finding that SliceGPT matches or exceeds SparseGPT 2:4 speedup at large model scales (Tables 12–13) while simultaneously reducing GPU count (Table 2) suggests that for the largest models, reduction of the embedding dimension may be a more effective strategy than introduction of weight sparsity — especially given that activation memory reduction enables larger batch sizes and fewer GPUs, benefits that weight-only sparsity cannot provide.

Follow-Up Research This Work Enables

Characterizing which capabilities are destroyed by dimension deletion — and which are preserved — across a broad capability spectrum. SliceGPT's evaluation uses WikiText-2 perplexity and five commonsense QA benchmarks. These are standard in the compression literature but do not test whether slicing selectively degrades specific capabilities: mathematical reasoning (GSM8K), factual knowledge (MMLU), code generation (HumanEval), few-shot in-context learning, long-form generation coherence, or multilingual performance. A systematic capability audit — slicing LLAMA-2 70B at 10%, 20%, and 30% and evaluating on 15–20 diverse benchmarks spanning reasoning, knowledge, generation, and code — would reveal whether PCA-guided dimension deletion preserves capabilities uniformly or whether some capabilities are disproportionately vulnerable. The spectral analysis (Figure 8) provides a mechanistic hypothesis: capabilities that rely on information distributed broadly across the embedding dimension (flat spectrum) will degrade more than capabilities concentrated in the leading principal components. A study that correlates per-task degradation with the eigenvalue spectrum at different layers would test this hypothesis and provide practitioners with predictive guidance on which tasks their sliced model can safely handle.

Combining SliceGPT with quantization to compound compression benefits. The paper notes in Section 5 that "complementary methods including quantization ... could be used" on top of sliced models, but no experiments are presented. A natural follow-up would apply GPTQ or AWQ quantization to already-sliced models, targeting a combined compression ratio: e.g., SliceGPT at 20% embedding reduction plus 4-bit quantization yields approximately $0.8 \times 0.25 = 0.20$ of the original memory footprint (since the 80%-size FP16 model is further reduced to 25% via quantization). The key question is whether the two compression axes interact: does slicing change the weight distribution in ways that make quantization more or less effective? The paper's transformation absorbs orthogonal matrices into the weights ($\tilde{W}_{\text{in}} = Q^\top W_{\text{in}}$), which could alter the per-channel variance and outlier structure that quantization methods are sensitive to. A study measuring quantization error (e.g., perplexity of GPTQ-applied-to-sliced models vs. GPTQ-applied-to-dense models at equivalent total compression) would determine whether SliceGPT can serve as a preprocessing step that improves quantizability by rotating to a numerically favorable basis — or whether the two methods interfere.

Training models to be sliceable from the start, using the invariance as a training objective. SliceGPT is a post-hoc method applied to already-trained models. But the computational invariance property exists at every step of training — the network is always in an equivalence class under orthogonal transformations. Could a model be trained to have a rapidly-decaying activation spectrum, making it more compressible by SliceGPT after training? A regularization term added during pretraining — penalizing the entropy of the eigenvalue distribution of $X_\ell^\top X_\ell$ at each layer, or encouraging the signal to concentrate in fewer principal components — would produce models that are "sliceable by design." This turns compression from a post-hoc fix into a training-time objective. The experiment would compare: (a) standard pretraining followed by SliceGPT vs. (b) spectrum-regularized pretraining followed by SliceGPT, measuring whether the regularized model achieves lower perplexity at the same slicing percentage. The paper's spectral analysis (Figure 8) showing that OPT naturally has faster spectral decay than LLAMA-2 suggests that architectural and training choices influence compressibility — making it plausible that explicit regularization could push this further.

Alternative transformation criteria beyond PCA for specialized objectives. PCA minimizes L2 reconstruction error of the activations, which is a natural criterion for preserving the signal's statistical structure but is not necessarily optimal for preserving specific downstream capabilities. The invariance theorem allows any orthogonal $Q$ to be absorbed. This opens the door to task-specific transformation criteria:

• Quantization-friendly rotations: Find $Q$ such that the transformed weights $\tilde{W} = Q^\top W$ have reduced per-channel variance or fewer outliers, improving post-training quantization quality. This would use the invariance to solve a different compression problem — not dimension reduction but numerical range reduction.
• Sparsity-inducing rotations: Find $Q$ such that the transformed weights have concentrated structure (many near-zero entries), enabling pruning after rotation with minimal information loss. This combines the invariance with traditional pruning, using the rotation as a pre-processing step that makes pruning less destructive.
• Hardware-aligned rotations: Find $Q$ such that the computational pattern of the transformed network better matches specific hardware characteristics (e.g., aligning the most active dimensions with the widest SIMD units, or structuring the residual matrices $Q_{\ell-1}^\top Q_\ell$ to have block-diagonal or low-rank structure that reduces their overhead).

Each of these would require defining an optimization objective over the orthogonal group $O(D)$ — a constrained optimization problem — and evaluating whether the resulting $Q$ produces better compression-quality tradeoffs than PCA for the specific downstream metric. The paper's framework makes this newly tractable because it provides the absorption rules (Theorem 1) that decouple the transformation choice from the network's functional behavior.

Stress-testing SliceGPT under distribution shift between calibration and deployment. The paper shows that calibration dataset choice affects downstream accuracy (Alpaca outperforms WikiText-2 by 5–8 accuracy points on zero-shot tasks; Figure 5). But it does not systematically characterize what happens when the deployment distribution diverges from the calibration distribution. A controlled study would: calibrate on data from domain A (e.g., Wikipedia), deploy on domains B (news), C (code), D (medical text), and E (non-English), and measure perplexity degradation as a function of domain distance. This would produce a "sensitivity curve" showing how quickly slicing quality degrades with distribution shift. The hypothesis is that PCA preserves dimensions carrying information about the calibration distribution's statistical structure; dimensions important for out-of-distribution text are more likely to be sliced. A strong negative result — e.g., a 25% sliced model showing 50% higher perplexity on code than on the calibration domain — would establish a critical practical limitation and motivate research on calibration-set robustness (e.g., using a mixture of diverse domains for calibration, or developing domain-agnostic transformation criteria).

Eliminating or reducing the residual connection overhead through structured approximations. The per-block matrices $Q_{\ell-1}^\top Q_\ell$ inserted into residual connections are necessary for the per-block extension of the invariance theorem (Section 3.3) but add a $D_{\text{small}} \times D_{\text{small}}$ matrix multiplication at every layer. The paper never quantifies this overhead or explores mitigation. A follow-up study would: (a) measure what fraction of total inference FLOPs are spent in these residual operations for different model sizes and slicing percentages, (b) evaluate whether $Q_{\ell-1}^\top Q_\ell$ can be approximated by a structured matrix (block-diagonal, low-rank, or Kronecker-factored) with minimal perplexity loss, and (c) test whether absorbing these matrices into adjacent operations (e.g., fusing with the preceding RMSNorm or the following attention projection) via custom CUDA kernels recovers additional speedup beyond what Table 2 reports. The paper's Hugging Face implementation uses separate matrix multiplications; a fused kernel that combines RMSNorm, residual transformation, and the first projection of the next block could substantially reduce kernel launch overhead and improve the latency numbers, particularly for batch-size-1 inference where kernel launch latency is a larger fraction of total time.

Practical Applications and Downstream Use Cases

Reducing GPU count for LLM deployment on consumer and edge hardware. The most direct application of SliceGPT is enabling large models to fit on fewer GPUs — or, for models at the boundary of single-GPU feasibility, enabling them to fit on a single consumer GPU. The paper's concrete numbers: LLAMA-2 70B at 25% slicing goes from requiring 7 Quadro RTX6000 GPUs (24GB each) to 5 — a 2-GPU reduction — while maintaining better perplexity than SparseGPT 2:4 (Table 2: 4.60 vs. 4.98) and the same per-operation speedup (Table 12: 1.32× vs. 1.32×). On A100 40GB GPUs, the requirement drops from 4 to 3. At 50% slicing (accepting a perplexity tradeoff), the largest models fit on a single H100 instead of two, with throughput increasing $3.75\times$ for LLAMA-2 70B and $6.26\times$ for OPT 66B (Table 11). For a startup or research lab with a fixed GPU budget, this directly translates to being able to serve a 70B-parameter model where previously only a 13B model would fit — a capability jump that does not require additional hardware investment. The deployment scenario is: take a pre-trained LLAMA-2 70B, run SliceGPT with Alpaca calibration in ~3.5 hours on an H100 (or rent one from a cloud provider for ~$10–15 at on-demand prices), apply optional recovery fine-tuning for ~1.5 hours, and deploy on 3× A100 instead of 4× — saving 25% on ongoing inference GPU costs indefinitely.

Batch inference pipelines with memory-constrained GPUs. For organizations running bulk inference — evaluating models on large test suites, generating synthetic training data, or processing document collections — the activation memory reduction from SliceGPT's dimension reduction enables larger batch sizes, directly increasing throughput. The paper's throughput benchmark (Table 11) shows that 25% sliced LLAMA-2 70B can double its batch size on H100 GPUs (128 → 256) because the $D_{\text{small}}$-dimensional activations consume less memory. Total throughput increases from 541 to 839 tokens/s on 2 GPUs (1.55×). In a batch processing pipeline processing billions of tokens, a 55% throughput improvement reduces total GPU-hours by 35%, which at cloud GPU pricing translates to substantial cost savings. This benefit compounds with the GPU count reduction: not only does each GPU process more tokens per second, but fewer GPUs are needed to hold the model. A deployment that required 8× A100 for the dense model might run the sliced model on 6× A100 with 1.5× per-GPU throughput, reducing total GPU requirements by approximately 55%.

On-device or single-GPU deployment of otherwise multi-GPU models. The paper's 50% slicing results, while acknowledging the perplexity degradation, demonstrate a regime where models that absolutely require multiple GPUs in their dense form become single-GPU-capable after slicing. OPT 66B at 50% slicing fits on a single H100 (vs. 2 for dense), and LLAMA-2 70B similarly drops from 2 GPUs to 1. For applications where the dense model's quality is overkill — e.g., internal prototyping, low-stakes text generation, or applications where latency and cost dominate accuracy considerations — aggressive slicing plus recovery fine-tuning could produce a model that fits within a single-GPU budget while retaining usable quality. The paper does not advocate this, but the data (Tables 9–10, 50% slicing not shown for RFT but implied by the throughput benchmarks) suggests a practical path: accept a perplexity degradation in exchange for halving the hardware requirement. This is especially relevant for deployments on edge servers, local inference on workstations, or scenarios where multi-GPU communication overhead dominates latency. The single-GPU deployment also simplifies infrastructure — no tensor parallelism, no pipeline parallelism, no inter-GPU communication — reducing the operational complexity of serving.

When to Prefer This Method

The paper does not explicitly articulate a decision rule positioning SliceGPT against named alternatives, and the primary baseline (SparseGPT 2:4) serves as a reference point rather than a direct competitor across all dimensions. SliceGPT's value is not that it uniformly dominates other compression methods, but that it introduces a new tradeoff dimension — embedding dimension reduction — whose benefits compound in specific deployment scenarios (GPU count reduction, batch size scaling) and whose costs vary by model family (spectral properties, RFT requirements). The paper's results support the following practical guidance, but the authors do not frame it as an explicit decision matrix:

• The strongest case for SliceGPT is compressing models at 30B+ parameters for deployment on memory-constrained hardware, where the GPU count reduction and activation memory savings provide benefits that weight-only sparsity cannot. The paper's LLAMA-2 70B results on consumer GPUs (7 → 5 GPUs at 25% slicing, Table 2) and the throughput scaling (1.55× at 25%, 3.75× at 50%, Table 11) make this case empirically.
• SliceGPT requires a calibration dataset, and downstream task performance depends on calibration-deployment alignment. Practitioners with clear, stable deployment distributions should calibrate on data representative of that distribution. The paper's Alpaca-vs-WikiText-2 comparison (Figure 5) shows this choice can matter by 5–8 accuracy points.
• SliceGPT works best on models whose activation spectra show rapid eigenvalue decay — the paper's spectrum analysis (Figure 8) provides a diagnostic: if the first ~50% of principal components capture >90% of total variance, slicing at 25% should be safe. OPT models exhibit this; LLAMA-2 models do so to a lesser extent, which is why LLAMA-2 requires recovery fine-tuning for competitive results.
• For small models (<7B parameters), SliceGPT at compression ratios that produce meaningful speedup (25%+) degrades accuracy substantially without recovery fine-tuning (LLAMA-2 7B drops from 69.00% to 55.48% at 25% slicing, Table 7). SparseGPT 2:4 or other methods may be preferable in this regime, particularly when GPU count reduction is not a binding constraint.