GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

ArXiv: 2403.03507

🎯 Pitch

You can pre-train a 7-billion-parameter language model on a single consumer GPU with just 24GB of memory—no model parallelism, no offloading. GaLore achieves this by projecting gradients into a periodically updated low-rank subspace, slashing optimizer memory by over 80% while matching full-rank training quality. The key insight is that gradients naturally exhibit a stable low-rank structure during training, which the paper proves for common architectures and verifies at scale.

---

1. Executive Summary

This paper proposes Gradient Low-Rank Projection (GaLore), a memory-efficient training strategy that enables full-parameter learning while using substantially less memory than conventional optimizers or low-rank adaptation methods like LoRA. Evaluated on pre-training LLaMA architectures (60M to 7B parameters) on the C4 dataset and fine-tuning RoBERTa on GLUE tasks, GaLore projects weight gradients into a periodically updated low-rank subspace (by applying SVD to the gradient matrix every ~200 iterations and tracking optimizer states for the compact representation rather than the full gradient), reducing optimizer state memory by up to 65.5% in BF16 and 82.5% in 8-bit mode while matching full-rank training perplexity. The approach establishes that pre-training a 7B model is feasible on a single consumer GPU with 24GB memory (e.g., NVIDIA RTX 4090) without model parallelism, checkpointing, or offloading—but only when the gradient's empirically observed low-rank structure holds throughout training, which the paper demonstrates both theoretically (for reversible networks and Transformer FFN layers) and empirically across model scales up to 19.7B training tokens.

2. Context and Motivation

The Core Problem: LLM Training Is Prohibitively Memory-Intensive

The fundamental challenge this paper addresses is straightforward to state but enormously consequential: training large language models requires far more GPU memory than most practitioners can access. The memory bottleneck is not just the model weights themselves—it is the entire optimization machinery that surrounds them. For a typical Adam optimizer, training a LLaMA 7B model requires storing:

• Model parameters: ~14 GB (in BF16 format)
• Gradient moment estimates (Adam states): ~28 GB — twice the parameter count since Adam maintains both first-moment ($M$) and second-moment ($V$) estimates for every parameter
• Weight gradients themselves: ~14 GB
• Activations: variable, typically 2+ GB depending on batch size

This adds up to at least 58 GB for a single batch (Section 1, Figure 1), which far exceeds the 24 GB available on consumer GPUs like the NVIDIA RTX 4090, and even strains datacenter GPUs like the 80 GB A100 when using reasonable batch sizes. The paper notes that these requirements have been partially addressed through engineering approaches — gradient checkpointing (Chen et al., 2016) trades compute for memory by recomputing activations during backward passes, and memory offloading (Rajbhandari et al., 2020) shuffles tensors between GPU and CPU memory — but these come with significant throughput penalties and add system complexity. There remains a critical gap: a principled, optimization-level approach that reduces the optimizer's memory footprint without sacrificing the ability to train all parameters.

Why This Problem Matters: More Than Just Enabling Consumer-GPU Training

The memory bottleneck has far-reaching implications beyond the appealing narrative of "train LLMs on your gaming GPU." The paper identifies several dimensions of impact (Section 1, Impact Statement):

Democratization of LLM research. When training a 7B model requires clusters of datacenter GPUs, only well-resourced industrial labs can experiment with architectural innovations, training recipes, or novel optimizers at scale. Memory-efficient training that works on consumer hardware lowers the barrier to entry and enables broader scientific exploration. This is not merely about individual practitioners — academic labs with limited GPU budgets could pre-train and study models at scales currently reserved for industry.

Environmental and carbon impact. The memory constraint forces practitioners toward distributed training on many GPUs, which increases total energy consumption through inter-GPU communication overhead and the fixed power costs of running more machines. A memory-efficient method that enables pre-training on fewer GPUs (or lower-end GPUs) directly reduces the carbon footprint of LLM development, which the paper explicitly frames as a goal.

Elastic and decentralized training. When model training fits within a single GPU's memory, data parallelism becomes the only communication pattern needed — gradients from different GPUs are averaged, but no model parameters need to be split across devices. Data-parallel communication has much lower bandwidth requirements than model parallelism (which splits individual layers across GPUs), opening the possibility of training across geographically distributed, low-bandwidth consumer hardware. The paper explicitly envisions "elastic data distributed training on low-bandwidth consumer-grade hardware" (Section 7).

Fine-tuning economics. While pre-training from scratch is the most dramatic use case, the memory bottleneck also affects fine-tuning — organizations that deploy many specialized models must store separate optimizer states for each fine-tuned variant. A method that works for both pre-training and fine-tuning (which GaLore does) provides a unified solution across the entire model lifecycle.

The Dominant Prior Approach: Low-Rank Adaptation (LoRA) and Its Limitations

The most influential memory-reduction technique in LLMs is LoRA (Hu et al., 2022), which has become the de facto standard for parameter-efficient fine-tuning. However, the paper identifies systematic limitations that prevent LoRA (and its variants) from being a general solution for memory-efficient pre-training.

How LoRA works. LoRA freezes the pre-trained weight matrix $W_0 \in \mathbb{R}^{m \times n}$ and learns an additive low-rank update $BA$, where $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$ with $r \ll \min(m, n)$. The full weight is $W = W_0 + BA$. During training, only $B$ and $A$ receive gradients, so the optimizer only needs to store states for $mr + nr$ parameters rather than $mn$. This dramatically reduces memory — but only when $W_0$ is already a good solution (as in fine-tuning). For pre-training from scratch, $W_0$ is random initialization, and the low-rank constraint fundamentally restricts what the model can learn.

Limitation 1: LoRA cannot match full-rank training performance. The paper cites evidence that "for fine-tuning, LoRA is not shown to reach a comparable performance as full-rank fine-tuning" (Xia et al., 2024), and the results in Table 2 show that for pre-training, LoRA's perplexity degrades severely as model size increases — from a modest gap at 60M parameters (LoRA: 34.99 vs. Full-Rank: 34.06) to catastrophic failure at 1B parameters (LoRA: 19.21 vs. Full-Rank: 15.56). This growing gap suggests that LoRA's low-rank constraint becomes increasingly incompatible with what large models need to learn.

Limitation 2: ReLoRA requires full-rank warmup. ReLoRA (Lialin et al., 2024) attempts to extend LoRA to pre-training by periodically merging the learned $BA$ adaptor back into $W$ and reinitializing new low-rank adaptors — essentially stitching together multiple low-rank updates across time. However, the paper notes that ReLoRA "requires a full-rank training warmup to achieve comparable performance as the standard baseline." Without this warmup phase (which uses full memory), performance degrades substantially. Table 2 shows ReLoRA at 1B achieves 18.33 — better than LoRA's 19.21 but still far from full-rank's 15.56. The requirement of a full-rank warmup defeats the purpose of memory reduction during a critical phase of training.

Limitation 3: Why low-rank weight constraints are fundamentally limiting. The paper provides theoretical motivation for why constraining weights to be low-rank is problematic (Section 3.2). Consider a simple linear regression $\mathbf{y} = W\mathbf{x}$: if the optimal $W^*$ has high rank (which is common when modeling complex dependencies between high-dimensional input and output spaces), then any low-rank parameterization can never reach the optimal solution, regardless of the optimizer used. More subtly, the paper argues that LoRA "alters the training dynamics" — when $W$ is represented as $W_0 + BA$, the gradient flow through $B$ and $A$ creates different optimization trajectories than training $W$ directly, potentially leading to worse local optima even when the optimal $W^*$ happens to be low-rank.

The Counterintuitive Alternative: Gradient Is Low-Rank, Not Weight

The paper's crucial observation — and what distinguishes it from the entire LoRA lineage — is that the gradient matrix $G \in \mathbb{R}^{m \times n}$ becomes low-rank during training, even though the weight matrix $W$ itself is not. This means that while $W$ may need its full $mn$ degrees of freedom to represent a good solution, the updates to $W$ at each step lie in a low-dimensional subspace. If you can identify and track that subspace, you can:

1. Project the gradient $G$ into the low-rank form $P^\top G Q$ (where $P \in \mathbb{R}^{m \times r}$, $Q \in \mathbb{R}^{n \times r}$)
2. Run your memory-hungry optimizer (e.g., Adam) on the compact $r \times r$ representation, keeping optimizer states of size $O(r(m+n))$ instead of $O(mn)$
3. Project the optimizer's update back to the original space: $\tilde{G} = P \cdot \text{Adam}(P^\top G Q) \cdot Q^\top$
4. Add the projected update to $W$ normally

The critical insight is that $W$ itself accumulates updates from many different low-rank subspaces over time (as the projections $P$ and $Q$ are periodically recomputed), allowing $W$ to eventually become full-rank — but the optimizer only ever needs to track statistics within whichever low-rank subspace is currently active. This is fundamentally different from LoRA, which constrains the weight to be low-rank throughout training.

Theoretical Basis for Gradient Low-Rankness

The paper does not simply assert that gradients become low-rank based on empirical observation — it provides theoretical justification for why this happens, which is essential for establishing GaLore as a principled method rather than a heuristic trick.

Reversible networks (Section 3.2). The paper proves that for a broad class of networks called "reversible networks" (Definition 3.1) — which includes linear layers without bias, ReLU activations, and polynomial activations, along with compositions like ResNet blocks — the gradient takes the specific parametric form:

$G_t = \frac{1}{N} \sum_{i=1}^N (A_i - B_i W_t C_i)$

where $A_i$, $B_i$, and $C_i$ depend on the input data and network architecture. $B_i$ and $C_i$ are positive semi-definite (PSD) matrices. This structure is not an assumption — Theorem 3.2 derives it directly from the chain rule applied to reversible networks for both $\ell_2$ loss and logsoftmax loss.

Why this structure leads to low-rank gradients. Lemma 3.3 proves that under vanilla SGD dynamics with this gradient form, the stable rank of $G_t$ (defined as $\|G_t\|_F^2 / \|G_t\|_2^2$, a continuous relaxation of matrix rank) satisfies:

$\text{sr}(G_t) \leq \text{sr}(G_{t_0}^\parallel) + \left(\frac{1 - \eta\lambda_2}{1 - \eta\lambda_1}\right)^{2(t-t_0)} \cdot \frac{\|G_0 - G_{t_0}^\parallel\|_F^2}{\|G_{t_0}^\parallel\|_2^2}$

where $\lambda_1 < \lambda_2$ are the two smallest distinct eigenvalues of the PSD matrix $S = \frac{1}{N}\sum_i C_i \otimes B_i$. The key term is the exponential decay factor $\left(\frac{1-\eta\lambda_2}{1-\eta\lambda_1}\right)^{2(t-t_0)}$ — since $\lambda_2 > \lambda_1$, this ratio is less than 1, driving the second term to zero exponentially fast. The gradient's stable rank converges to $\text{sr}(G_{t_0}^\parallel)$, which is bounded by the rank of the projection of the gradient onto the minimal eigenspace of $S$.

What this means intuitively. The gradient dynamics naturally suppress components of the gradient that correspond to larger eigenvalues of $S$, while components corresponding to the smallest eigenvalue $\lambda_1$ survive. If the eigenspace for $\lambda_1$ is low-dimensional (which Corollaries 3.4 and 3.5 show happens under mild conditions — for example, when the input features $\{\mathbf{f}_i\}$ don't span the whole space, or when the minimal eigenspace is 1-dimensional), then the gradient becomes provably low-rank during training.

Extension to Transformers (Appendix B.3). Recognizing that Transformers are not strictly reversible networks, the paper separately proves gradient low-rankness for the feed-forward network (FFN) layer of Transformers using the JoMA framework (Tian et al., 2024). The proof shows that the rows of the project-up weight matrix $V$ grow at different rates — the row corresponding to the largest entry of a certain data-dependent matrix $\mu$ grows exponentially faster than others, causing the gradient to become "exponentially more low-rank than $V$ itself" (Lemma B.3). While this proof covers only FFN layers (not attention), it provides theoretical grounding for the empirical observation that GaLore works well across full Transformer architectures.

Positioned Between Two Extremes

The paper positions GaLore carefully within the existing landscape:

• Against full-rank training: GaLore provides comparable convergence but with dramatically lower memory. It is not claiming to beat full-rank training in quality — the goal is to match it while using a fraction of the memory. Table 2 shows GaLore achieving 15.64 perplexity vs. full-rank's 15.56 at 1B scale, a negligible gap.

• Against LoRA and ReLoRA: GaLore fundamentally differs in what it constrains to be low-rank. LoRA constrains weights — which limits expressiveness. GaLore constrains gradients — which does not limit the final weight matrix's rank. The paper demonstrates this difference empirically: at 1B scale, GaLore (15.64 perplexity) dramatically outperforms both LoRA (19.21) and ReLoRA (18.33). The theoretical analysis in Section 3.2 provides the explanatory framework: the gradient form that arises from reversible networks naturally converges to low rank, while nothing forces the optimal $W$ to be low-rank.

• Against pure engineering solutions (offloading, checkpointing): These are complementary, not competing. GaLore can be combined with 8-bit optimizers (Section 5.2), per-layer weight updates (Section 4.3), and activation checkpointing (Section 5.5) for cumulative memory savings. The paper demonstrates this by achieving training of LLaMA 7B in 22 GB.

The paper's strongest positioning claim is that GaLore is the first method to enable full-parameter pre-training at 7B scale on a single consumer GPU — not through a new distributed training scheme or compression trick, but by recognizing and exploiting a fundamental mathematical property of how neural network gradients evolve. This is what the executive summary captures as the paper's central contribution, and the full theoretical and empirical apparatus of Sections 3–5 is built to substantiate it.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This is primarily a theoretical analysis and empirical optimization paper whose core idea is that we can dramatically reduce the memory footprint of training large neural networks without sacrificing model quality, by observing that weight gradients become low-rank during training and manipulating them in a compressed subspace rather than in their full, memory-hungry form.

The paper designs a memory-efficient training strategy that projects weight gradients into a low-rank subspace using periodically updated SVD-based projection matrices, runs the optimizer (e.g., Adam) on the compact projected representation to accumulate gradient statistics, then projects the resulting update back to the original parameter space — enabling full-parameter learning while storing optimizer states for only the low-rank core rather than the full gradient tensor.

3.2 Big-Picture Architecture (Diagram in Words)

The GaLore system has four major components that interact during each training step:

1. Base Model and Forward Pass — the standard neural network (e.g., LLaMA Transformer) performs a forward pass and computes the loss $\varphi(W_t)$. This component is unchanged from conventional training.

2. Gradient Computation — backpropagation computes the weight gradient matrix $G_t = -\nabla_W \varphi_t(W_t) \in \mathbb{R}^{m \times n}$ for each layer's weight matrix $W$. At this point, $G_t$ is full-rank and would normally be passed directly to the optimizer.

3. Projection Matrix Manager — every $T$ iterations (e.g., $T = 200$), this component computes the singular value decomposition (SVD) of the current gradient $G_t$, extracts the top-$r$ left and right singular vectors to form projection matrices $P \in \mathbb{R}^{m \times r}$ and $Q \in \mathbb{R}^{n \times r}$, and stores them. Between recomputations, the same $P$ and $Q$ are reused.

4. Compact Optimizer — rather than tracking optimizer states for the full $m \times n$ gradient, the gradient is first projected to a compact $r \times r$ representation $R_t = P^\top G_t Q$ (if $m \leq n$, only one projection is used: $R_t = P^\top G_t \in \mathbb{R}^{r \times n}$). The optimizer (e.g., Adam) maintains its first-moment and second-moment buffers in this compact space, consuming $O(r \times n)$ memory instead of $O(m \times n)$. After computing the update $N_t = \text{Adam}(R_t)$, it is projected back to the original space as $\tilde{G}_t = P \cdot N_t$ and added to the weight $W_{t+1} = W_t + \eta \tilde{G}_t$.

Information flows: forward pass $\rightarrow$ loss $\rightarrow$ backward pass $\rightarrow$ full gradient $G_t$ $\rightarrow$ (if $t \bmod T = 0$) compute SVD, update $P$ $\rightarrow$ project $G_t$ to compact $R_t$ $\rightarrow$ run Adam in compact space, producing $N_t$ $\rightarrow$ project back to $\tilde{G}_t = P \cdot N_t$ $\rightarrow$ update $W_{t+1} = W_t + \eta \tilde{G}_t$.

3.3 Roadmap for the Deep Dive

• First, the theoretical foundation: why gradients become low-rank during training. This is essential because the entire method rests on the claim that gradients (not weights) are low-rank — I will explain the gradient form for reversible networks, the key lemma proving exponential convergence to low stable rank, and the extension to Transformer FFN layers.
• Second, the GaLore update rule (Definition 3.6) and how it differs from both full-rank training (Equation 1) and LoRA-style low-rank updates (Equation 5). This defines the mathematical operator at the heart of the method.
• Third, the convergence theory (Theorem 3.8) that guarantees GaLore with fixed projections converges under the same gradient structure, establishing that the projection does not break optimization.
• Fourth, the multi-subspace composition mechanism (Section 4.1): how periodic SVD recomputation allows GaLore to switch subspaces, enabling full-rank weight learning while maintaining low memory throughout.
• Fifth, the memory-efficient optimization integration (Section 4.2–4.3): how GaLore plugs into Adam, 8-bit Adam, and Adafactor; the single-projection optimization (Algorithm 2); and how it combines with per-layer weight updates for additional savings.
• Sixth, the hyperparameter design (Section 4.4): the rank $r$, subspace change frequency $T$, and scale factor $\alpha$ — what each controls and why they exist.

3.4 Detailed, Sentence-Based Technical Breakdown

---

The Gradient Form That Makes Low-Rank Structure Possible

Before explaining GaLore itself, the paper establishes why weight gradients become low-rank during training. This theoretical foundation is not just mathematical window-dressing — it provides the justification for applying low-rank compression to gradients specifically, rather than to weights (as LoRA does) or to optimizer states (as Adafactor does).

Definition 3.1: Reversible Networks. A network $\mathcal{N}$ mapping input $\mathbf{x}$ to output $\mathbf{y} = \mathcal{N}(\mathbf{x})$ is called reversible if there exists a matrix $L(\mathbf{x}; W)$ such that $\mathbf{y} = L(\mathbf{x}; W) \mathbf{x}$, and the backpropagated gradient $\mathbf{g_x}$ satisfies $\mathbf{g_x} = L^\top(\mathbf{x}; W) \mathbf{g_y}$, where $\mathbf{g_y}$ is the backpropagated gradient at the output.

In plain language: a reversible network is one where both the forward pass and the backward pass can be expressed as linear transformations (though the transformation matrix itself depends on the input and weights in potentially nonlinear ways). This property holds for linear layers without bias, ReLU activations, leaky ReLU, and polynomial activations. Importantly, if $\mathcal{N}_1$ and $\mathcal{N}_2$ are reversible, then both their parallel combination $\alpha_1 \mathcal{N}_1(\mathbf{x}) + \alpha_2 \mathcal{N}_2(\mathbf{x})$ and their sequential composition $\mathcal{N}_2(\mathcal{N}_1(\mathbf{x}))$ are also reversible. This means ResNet blocks (which are $x + \mathcal{N}(x)$) are reversible when $\mathcal{N}$ uses bias-free layers and reversible activations — which covers most practical architectures.

Theorem 3.2: Gradient Form of Reversible Models. For a chained reversible network and a linear layer $l$ with weight matrix $W_l$, the gradient $G_l$ takes a specific parametric structure. Consider batch size $N = 1$ for clarity. For $\ell_2$ loss $\varphi := \frac{1}{2}\|\mathbf{y} - \mathbf{f}_L\|_2^2$:

$G_l = (J_l^\top \mathbf{y} - J_l^\top J_l W_l \mathbf{f}_{l-1}) \mathbf{f}_{l-1}^\top$

where $J_l := \text{Jacobian}(\mathcal{N}_L) \cdots \text{Jacobian}(\mathcal{N}_{l+1})$ is the product of all Jacobians after layer $l$, and $\mathbf{f}_{l-1} := \mathcal{N}_{l-1}(\dots \mathcal{N}_1(\mathbf{x}))$ is the input to layer $l$.

What this form represents structurally. The gradient is the outer product of two vectors: the first is the backpropagated error signal $(J_l^\top \mathbf{y} - J_l^\top J_l W_l \mathbf{f}_{l-1})$ (the difference between the target and the current prediction transformed backward through the network), and the second is the layer input $\mathbf{f}_{l-1}$. In matrix form, the first term $J_l^\top \mathbf{y} \mathbf{f}_{l-1}^\top$ is rank-1 (it is an outer product). The second term $J_l^\top J_l W_l \mathbf{f}_{l-1} \mathbf{f}_{l-1}^\top$ involves the current weight $W_l$, multiplied by PSD matrices on both sides (since $J_l^\top J_l$ and $\mathbf{f}_{l-1} \mathbf{f}_{l-1}^\top$ are PSD).

For the $K$-way logsoftmax loss (the standard loss for language modeling), Theorem 3.2(b) derives an analogous form:

$G_l = \left(J_l P_{\mathbf{1}}^\perp \mathbf{y} - \gamma K^{-1} J_l^\top P_{\mathbf{1}}^\perp J_l W_l \mathbf{f}_{l-1}\right) \mathbf{f}_{l-1}^\top$

where $P_{\mathbf{1}}^\perp := I - \frac{1}{K}\mathbf{1}\mathbf{1}^\top$ is the zero-mean projection matrix (removes the global offset from logits, since adding a constant to all logits doesn't change softmax probabilities), $\mathbf{y}$ is a one-hot label vector, and $\gamma \approx 1$ when logits are small.

Generalizing to batch size $N$. With $N$ independent samples, the gradient is the average:

$G_t = \frac{1}{N} \sum_{i=1}^N (A_i - B_i W_t C_i)$

where $A_i$, $B_i$, and $C_i$ are data-dependent matrices, and crucially $B_i$ and $C_i$ are positive semi-definite (PSD). For the $\ell_2$ case: $A_i = J_{l,i}^\top \mathbf{y}_i \mathbf{f}_{l-1,i}^\top$, $B_i = J_{l,i}^\top J_{l,i}$, and $C_i = \mathbf{f}_{l-1,i} \mathbf{f}_{l-1,i}^\top$.

Why this parametric form matters. This structure — a sum of terms each involving the current weight $W_t$ multiplied by PSD matrices on both sides — is what makes the gradient become low-rank. The PSD property of $B_i$ and $C_i$ means that the linear operator $\mathcal{S}(G) := \frac{1}{N}\sum_i B_i G C_i$, when vectorized as $\text{vec}(\mathcal{S}(G)) = S \cdot \text{vec}(G)$ with $S = \frac{1}{N} \sum_i C_i \otimes B_i$, is also PSD. The eigenvalues of $S$ determine how different directions in gradient space evolve over time.

---

Lemma 3.3: Gradient Becomes Low-Rank During Training

The paper then proves that when the gradient has the form $G_t = \frac{1}{N}\sum_i (A_i - B_i W_t C_i)$ with constant $A_i$, $B_i$, $C_i$ after some time $t_0$, and when training with vanilla SGD ($W_t = W_{t-1} + \eta G_{t-1}$), the stable rank of $G_t$ satisfies:

$\text{sr}(G_t) \leq \text{sr}(G_{t_0}^\parallel) + \left(\frac{1 - \eta\lambda_2}{1 - \eta\lambda_1}\right)^{2(t-t_0)} \cdot \frac{\|G_0 - G_{t_0}^\parallel\|_F^2}{\|G_{t_0}^\parallel\|_2^2}$

where $S := \frac{1}{N}\sum_{i=1}^N C_i \otimes B_i$ is the Kronecker-sum PSD matrix, $\lambda_1 < \lambda_2$ are its two smallest distinct eigenvalues, $\eta$ is the learning rate, $G_{t_0}^\parallel$ is the projection of $G_{t_0}$ onto the minimal eigenspace $\mathcal{V}_1$ of $S$ (the subspace corresponding to eigenvalue $\lambda_1$), and $\text{sr}(X) := \|X\|_F^2 / \|X\|_2^2$ is the stable rank (a continuous measure of rank that is small when most of the matrix's energy is concentrated in a few singular vectors).

What this inequality means operationally. The stable rank of the gradient is bounded by two terms. The first term $\text{sr}(G_{t_0}^\parallel)$ is the stable rank of the gradient projected onto the slowest-decaying subspace — this is the "asymptotic" rank that remains after long training. The second term involves the coefficient $\left(\frac{1-\eta\lambda_2}{1-\eta\lambda_1}\right)^{2(t-t_0)}$, which decays exponentially to zero because $\lambda_2 > \lambda_1$ implies $0 < \frac{1-\eta\lambda_2}{1-\eta\lambda_1} < 1$. This means that any component of the gradient not aligned with $\mathcal{V}_1$ is exponentially suppressed over time, driving the gradient into the low-dimensional subspace $\mathcal{V}_1$.

Why this form (the vectorized gradient dynamics). The proof converts the matrix gradient update $G_t = G_{t-1} - \frac{\eta}{N}\sum_i B_i G_{t-1} C_i$ into a vectorized form $g_t = (I - \eta S) g_{t-1}$ using the Kronecker product identity $\text{vec}(BGC) = (C^\top \otimes B)\text{vec}(G)$. This makes the dynamics a standard linear recurrence, whose convergence rate in each eigendirection is governed by the corresponding eigenvalue of $S$. This is why the eigenstructure of $S$ controls the rank reduction — directions with larger eigenvalues decay faster, and only the subspace with the smallest eigenvalue survives.

The constant assumption caveat. The proof assumes $A_i$, $B_i$, $C_i$ become constant after time $t_0$. This is an approximation that holds when the network's representations and Jacobians stabilize, which happens in practice during certain phases of training (e.g., after initial rapid changes). The paper acknowledges that if $t_0 = 0$ and the matrices are constant from the start, $G_{t_0}^\parallel$ could be zero (a degenerate case), but this does not occur in practical training.

Corollary 3.4: Concrete low-rank bound. If $G_t$ takes the form $G_t = \frac{1}{N}\sum_i (\mathbf{a}_i - B_i W_t \mathbf{f}_i)\mathbf{f}_i^\top$ with full-rank $B_i$, and the set of feature vectors $\{\mathbf{f}_i\}$ has rank $N' < n$ (they don't span the full $n$-dimensional space), then $\text{sr}(G_{t_0}^\parallel) \leq n - N'$, and thus $\text{sr}(G_t) \leq n/2$ for large $t$. When $N'$ is small (few distinct feature directions), the gradient is a sum of $N'$ rank-1 updates and is naturally low-rank. When $N'$ is large (close to $n$), the null space $\mathcal{V}_1$ is small, which also forces low-rank behavior.

Corollary 3.5: Rank-1 special case. If the minimal eigenspace $\mathcal{V}_1$ is 1-dimensional with a decomposable eigenvector $\mathbf{v} = \mathbf{y} \otimes \mathbf{z}$ (the Kronecker product of two vectors), then $\text{sr}(G_{t_0}^\parallel) = 1$ and the gradient becomes rank-1. This is the most extreme case of gradient compression.

Extension to Transformer FFN layers (Appendix B.3). Using the JoMA framework, the paper separately proves that the gradient of the project-up weight in the feed-forward network becomes "exponentially more low-rank" than the weight itself. Lemma B.3 shows that when embeddings are fixed and column-orthonormal, and activations are linear, the transformed weight dynamics cause different rows to grow at dramatically different rates. The row corresponding to the largest entry of a data-dependent matrix $\mu$ grows to infinity at a critical time, while other rows remain finite, forcing the gradient toward rank-1. This provides theoretical coverage for at least the FFN component of Transformers.

---

Definition 3.6: The GaLore Update Rule

With the theoretical foundation established, the paper formally defines GaLore. The standard full-rank training update (Equation 1) is:

$W_T = W_0 + \eta \sum_{t=0}^{T-1} \tilde{G}_t = W_0 + \eta \sum_{t=0}^{T-1} \rho_t(G_t)$

where $G_t = -\nabla_W \varphi_t(W_t) \in \mathbb{R}^{m \times n}$ is the raw gradient, $\rho_t$ is an entry-wise stateful gradient regularizer (e.g., Adam), and $\tilde{G}_t = \rho_t(G_t)$ is the processed gradient actually added to the weights. For Adam, $\rho_t$ requires storing $M_t, V_t \in \mathbb{R}^{m \times n}$ (the first and second moment estimates), which costs $2mn$ memory in addition to the weight itself — tripling the parameter memory.

GaLore replaces this with a projected update:

$W_T = W_0 + \eta \sum_{t=0}^{T-1} \tilde{G}_t, \quad \tilde{G}_t = P_t \cdot \rho_t(P_t^\top G_t Q_t) \cdot Q_t^\top$

where $P_t \in \mathbb{R}^{m \times r}$ and $Q_t \in \mathbb{R}^{n \times r}$ are projection matrices (each with orthonormal columns, so $P_t^\top P_t = I_r$ and $Q_t^\top Q_t = I_r$), and $r \ll \min(m, n)$ is the projection rank.

What this computes step-by-step. First, the full gradient $G_t$ is compressed by left-multiplying by $P_t^\top$ (reducing rows from $m$ to $r$) and right-multiplying by $Q_t$ (reducing columns from $n$ to $r$), producing a compact $r \times r$ matrix $R_t = P_t^\top G_t Q_t$. Second, the optimizer $\rho_t$ is applied to this compact representation — critically, Adam's moment estimates now live in $\mathbb{R}^{r \times r}$ instead of $\mathbb{R}^{m \times n}$. Third, the resulting $r \times r$ update $N_t = \rho_t(R_t)$ is projected back to the original space by left-multiplying by $P_t$ (expanding rows from $r$ to $m$) and right-multiplying by $Q_t^\top$ (expanding columns from $r$ to $n$), yielding $\tilde{G}_t \in \mathbb{R}^{m \times n}$ which is added to $W_t$.

Why this form — and how it differs from LoRA. In LoRA (Equation 5), the update is $W_T = W_0 + B_T A_T$ where $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$ are the trainable parameters. The critical difference is that LoRA constrains the accumulated weight change to be low-rank: the matrix $B_T A_T$ has rank at most $r$, meaning $W_T$ can only deviate from $W_0$ by a low-rank matrix. In GaLore, the per-step gradient $\tilde{G}_t$ is projected through a low-rank bottleneck, but because $P_t$ and $Q_t$ can change over time (by recomputing SVD), the sum $\sum_t \tilde{G}_t$ can become full-rank. When $r = \min(m, n)$, GaLore with $\rho_t \equiv 1$ (identity regularizer) exactly recovers full-rank SGD, since $P_t P_t^\top G_t Q_t Q_t^\top = G_t$ when $P_t$ and $Q_t$ are full orthogonal matrices. LoRA never recovers full-rank training regardless of $r$, because the parameterization $W_0 + BA$ fundamentally constrains the weight matrix.

The projection strategy. Theorem 3.8 (discussed below) shows that for fastest convergence, $P$ and $Q$ should project onto the subspaces corresponding to the largest eigenvalues of the PSD matrices $B_i$ and $C_i$ (which appear in the gradient form). Since these matrices are parts of Jacobians and expensive to compute, the paper instead uses the SVD of $G_t$ itself as a proxy. At each recomputation step:

$G_t = U S V^\top \approx \sum_{i=1}^r s_i u_i v_i^\top$

$P_t = [u_1, u_2, \dots, u_r], \quad Q_t = [v_1, v_2, \dots, v_r]$

The top-$r$ left singular vectors form $P_t$, and the top-$r$ right singular vectors form $Q_t$. This captures the directions of largest variation in the current gradient, which empirically correlate with the important subspaces of $B_i$ and $C_i$.

---

Theorem 3.8: Convergence of GaLore with Fixed Projections

Having defined the update rule, the paper proves that GaLore converges under the same gradient structure that motivated it. The theorem assumes: the gradient has the form of Equation 8 ($G_t = \frac{1}{N}\sum_i(A_i - B_i W_t C_i)$), the matrices $A_i$, $B_i$, $C_i$ have Lipschitz continuity with respect to $W$ (constants $L_A$, $L_B$, $L_C$), the weight norm is bounded ($\|W_t\| \leq D$), and the projections are held constant: $P_t = P$, $Q_t = Q$. Under vanilla SGD ($\rho_t \equiv 1$), the projected gradient $R_t := P^\top G_t Q$ satisfies:

$\|R_t\|_F \leq \left[1 - \eta(\kappa_{t-1} - L_A - L_B L_C D^2)\right] \|R_{t-1}\|_F$

where $\kappa_t := \frac{1}{N}\sum_i \lambda_{\min}(P^\top B_i(W_t) P) \cdot \lambda_{\min}(Q^\top C_i(W_t) Q)$.

What this inequality means. The norm of the projected gradient contracts by a factor of at most $1 - \eta(\kappa_{t-1} - L_A - L_B L_C D^2)$ each step. If $\min_t \kappa_t > L_A + L_B L_C D^2$ (the smallest eigenvalue of the projected PSD matrices is large enough to overcome the Lipschitz constants), then the contraction factor is strictly less than 1, and $R_t \to 0$ — meaning GaLore converges to a stationary point in the projected subspace.

Why $\kappa_t$ controls the convergence rate. The term $\kappa_t$ is the average product of the minimum eigenvalues of the projected PSD matrices. When $P$ and $Q$ capture the top eigenvectors of $B_i$ and $C_i$, these minimum eigenvalues are large (since we keep the directions with large eigenvalues and discard those with small ones). This makes the contraction factor small and convergence fast. If $P$ and $Q$ were random projections that accidentally captured directions with near-zero eigenvalues of $B_i$ and $C_i$, $\kappa_t$ would be small, and convergence might stall — the optimizer would make little progress because the gradient in the projected subspace would barely respond to weight changes.

The Lipschitz penalty terms. $L_A$, $L_B$, and $L_C$ represent how much $A_i$, $B_i$, and $C_i$ change as the weights change. The combined term $L_A + L_B L_C D^2$ acts as a "drag" on convergence — even in the best subspace, the changing gradient structure prevents perfect contraction. The $D^2$ factor means that larger weight norms amplify the effect of changes in $B_i$ and $C_i$, making convergence harder for models with large weights (which can be mitigated by weight decay).

How this connects to the projection choice. The theorem reveals why SVD of $G_t$ is a reasonable proxy for the ideal projection (onto top eigenvectors of $B_i$ and $C_i$). Since $G_t$ itself is formed from $B_i$ and $C_i$ acting on $W_t$, the dominant singular vectors of $G_t$ tend to align with the dominant eigenvectors of $B_i$ and $C_i$. The paper acknowledges this is approximate — the ideal would be to directly compute the eigenstructure of $B_i$ and $C_i$ — but SVD of $G_t$ is computationally feasible (only done every $T$ steps) and empirically effective.

---

Section 4.1: Composition of Low-Rank Subspaces

The convergence theory assumes fixed projections $P$ and $Q$, but fixed projections would constrain the weight $W_t$ to only grow within a single low-rank subspace — contradicting the goal of full-parameter learning. The paper's solution is to switch subspaces periodically.

The multi-subspace weight update:

$W_t = W_0 + \Delta W_{T_1} + \Delta W_{T_2} + \dots + \Delta W_{T_n}$

where for $t$ in the $i$-th interval $[\sum_{j=1}^{i-1} T_j, \sum_{j=1}^i T_j]$, the projections $P$ and $Q$ are fixed, and $\Delta W_{T_i} = \eta \sum_{\tau=0}^{T_i-1} \tilde{G}_\tau$ is the accumulated update within that subspace. At the boundary between intervals (every $T$ steps), the projections are recomputed by running SVD on the current gradient $G_t$ and extracting the new top-$r$ singular vectors.

What this enables. Over time, $W_t$ can accumulate updates in many different low-rank subspaces, allowing it to become a full-rank matrix even though each individual update $\tilde{G}_\tau$ lies in a rank-$r$ subspace. This is illustrated in Figure 2: each colored arrow represents the accumulated update $\Delta W_{T_i}$ within one subspace, and the final weight is the sum of updates from multiple subspaces.

The subspace change frequency $T$ as a hyperparameter. Changing too frequently has several costs: (1) computing SVD is an $O(\min(m,n) \cdot mn)$ operation that adds overhead (the paper reports $<10\%$ overhead with $T = 200$); (2) frequent changes violate the constant-projection assumption of Theorem 3.8, potentially harming convergence in each individual subspace; (3) the optimizer states (Adam's $M$ and $V$) accumulate over multiple steps and resetting projections disrupts their accumulated statistics. Changing too infrequently risks the optimization stalling in a subspace that is no longer the most important direction for reducing loss — Theorem 3.8 guarantees good progress within the projected subspace, but the overall loss may require exploring different directions.

The sweet spot. The paper finds empirically (Figure 5, left) that values between $T = 50$ and $T = 1000$ make "no much difference," and uses $T = 200$ across all experiments. This is long enough that SVD overhead is amortized over many steps (200 iterations of Adam updates cost far more than one SVD), and short enough that the projections adapt to the changing gradient structure before convergence in any one subspace stalls.

---

Section 4.2: Memory-Efficient Optimization — Algorithm 2

Algorithm 2 (reproduced in the paper) shows how GaLore integrates with Adam for a single layer weight $W \in \mathbb{R}^{m \times n}$ with $m \leq n$. The key design choice is using only one projection matrix $P \in \mathbb{R}^{m \times r}$ rather than two, reducing the gradient from $m \times n$ to $r \times n$:

$R_t = P_t^\top G_t \in \mathbb{R}^{r \times n}$

The Adam optimizer then tracks first and second moment estimates $M_t, V_t \in \mathbb{R}^{n \times r}$ (note: transposed to $\mathbb{R}^{n \times r}$ for efficient memory layout, as the paper states in the initialization $M_0, V_0 \in \mathbb{R}^{n \times r}$).

Step-by-step through Algorithm 2:

1. Check if subspace change is due: if $t \bmod T = 0$, compute SVD of $G_t$ (size $m \times n$) and set $P_t$ to the first $r$ left singular vectors $U[:, :r]$. Otherwise, reuse $P_t = P_{t-1}$.

2. Project gradient: $R_t = P_t^\top G_t$, producing an $r \times n$ compact representation.

3. Run Adam in compact space (standard Adam update, but on $R_t$ instead of $G_t$):

   • $M_t = \beta_1 M_{t-1} + (1 - \beta_1) R_t$
   • $V_t = \beta_2 V_{t-1} + (1 - \beta_2) R_t^2$ (element-wise square)
   • Bias correction: $\hat{M}_t = M_t / (1 - \beta_1^t)$, $\hat{V}_t = V_t / (1 - \beta_2^t)$
   • $N_t = \hat{M}_t / (\sqrt{\hat{V}_t} + \epsilon)$ (element-wise division)

4. Project back: $\tilde{G}_t = \alpha \cdot P_t N_t$, where $\alpha$ is the scale factor (a hyperparameter), and $P_t N_t$ maps the $r \times n$ compact update back to $m \times n$.

5. Weight update: $W_t = W_{t-1} + \eta \cdot \tilde{G}_t$.

Why the single-projection design. When $m \leq n$, projecting only the rows (via $P$) is sufficient because the column space of $G_t$ is already captured in the $r \times n$ representation. Using only $P$ (not both $P$ and $Q$) reduces the projection matrix memory from $(m + n)r$ to $m r$, which matters for wide layers where $n$ is large (e.g., the vocabulary projection in language model heads). The memory comparison in Table 1 shows GaLore requires $mn + mr + 2nr$ total memory (weights + projection + optimizer states in compact form), while LoRA requires $mn + 3mr + 3nr$ (weights + adaptor parameters + adaptor optimizer states + frozen weight).

Why reorder to $\mathbb{R}^{n \times r}$ for optimizer states. The paper initializes $M_0, V_0 \in \mathbb{R}^{n \times r}$ rather than $\mathbb{R}^{r \times n}$. This is a memory layout optimization — when $n \gg r$, storing the states as $n \times r$ means the leading dimension is $n$, which is contiguous with the $n$ columns of the projected gradient and improves memory access patterns during Adam's element-wise operations.

The projection matrix memory is not trivial but is smaller than the savings. For a layer with $m = n = 4096$ (typical for 7B models) and rank $r = 1024$, the projection matrix $P$ requires $4096 \times 1024 \approx 4.2\text{M}$ parameters (8.4 MB in BF16). This is small compared to the optimizer state savings: full Adam states require $2 \times 4096^2 \approx 33.6\text{M}$ parameters (67.2 MB), while compact Adam states require $2 \times 4096 \times 1024 \approx 8.4\text{M}$ parameters (16.8 MB), a net saving of 50.4 MB per layer.

---

Section 4.3: Combining with Existing Memory-Efficient Techniques

GaLore is designed to compose with other memory-saving approaches, and the paper integrates two specifically.

8-bit optimizers (Dettmers et al., 2022). The standard 8-bit Adam quantizes the optimizer states $M$ and $V$ from 32-bit floating point to 8-bit integers using block-wise quantization, reducing state memory by roughly 4×. GaLore already reduces the number of optimizer state entries from $mn$ to $(m+n)r$; stacking 8-bit quantization on top further reduces the size per entry. The paper calls this combination "8-bit GaLore" and uses it as the default for the 7B experiments. Concretely, in 8-bit GaLore, the compact optimizer states $M_t, V_t \in \mathbb{R}^{n \times r}$ are stored in 8-bit format instead of 32-bit, cutting their memory from $2 \times n \times r \times 4$ bytes to $2 \times n \times r \times 1$ byte.

Per-layer weight updates (Lv et al., 2023a,b). Normally, backpropagation computes gradients for all layers and stores them in memory, then the optimizer updates all weights at once. This requires keeping all weight gradients simultaneously. Per-layer weight updates instead interleave: immediately after computing the gradient for layer $l$, run the optimizer update for layer $l$ and discard the gradient before computing gradients for layer $l-1$. This eliminates the need to store the full set of weight gradients. GaLore with per-layer weight updates is denoted as "8-bit GaLore (layer-wise)" in Figure 4 and saves an additional 13.5 GB for the 7B model compared to 8-bit Adam (because 8-bit Adam must still store full weight gradients).

The tradeoff. Per-layer updates reduce peak memory (only one layer's gradient is alive at any time) but prevent some parallelism in the backward pass. The paper reports that disabling per-layer updates (retaining gradients) increases throughput from 1019 to 1109 tokens/second for the 1B model (Table 11), an 8.8% improvement, at the cost of higher memory.

---

Section 4.4: Hyperparameters of GaLore

GaLore introduces three hyperparameters beyond the base optimizer's settings (e.g., Adam's $\beta_1, \beta_2, \epsilon$):

Rank $r$. This is the same hyperparameter as in LoRA, controlling the dimensionality of the projected subspace. Larger $r$ means more of the gradient is preserved, yielding faster convergence per step but higher memory. The paper sweeps $r$ from 128 to 1024 depending on model size (Table 2): $r = 128$ for 60M, $r = 256$ for 130M and 350M, and $r = 512$ or $1024$ for 1B and 7B. Figure 5 (right) shows that training with a smaller rank ($r = 128$) for more steps can achieve lower loss than training with a larger rank ($r = 512$) for fewer steps, suggesting a memory-compute tradeoff: in severely memory-constrained settings, reduce $r$ to stay within budget and train longer to compensate.

Subspace change frequency $T$. As discussed in Section 4.1, this controls how often the SVD is recomputed. The paper uses $T = 200$ across all experiments after an ablation (Figure 5, left) showing that values from 50 to 1000 work similarly. The paper explicitly notes that "for small $r$, the subspace switching should happen more to avoid wasting optimization steps in the wrong subspace, while for large $r$ the gradient updates cover more subspaces, providing more cushion" — meaning the optimal $T$ depends on $r$, though the paper does not jointly optimize them.

Scale factor $\alpha$. This controls the strength of the low-rank update, multiplying the projected-back gradient before adding it to the weight: $\tilde{G}_t = \alpha \cdot P_t N_t$. The paper distinguishes this from LoRA's scaling: "$\alpha$ does not depend on the rank $r$ in our case. This is because, when $r$ is small during pre-training, $\alpha/r$ significantly affects the convergence rate, unlike fine-tuning." In LoRA, the scaling is typically $\alpha/r$ to keep the update magnitude invariant to rank, but GaLore uses a fixed $\alpha = 0.25$ across all model sizes (Table 5, Appendix C.1). Since $\alpha$ can be viewed as a fractional multiplier on the learning rate, the "actual learning rate" for most modules becomes $\eta \cdot \alpha = 0.01 \times 0.25 = 0.0025$, which the paper notes is "still, a relatively large stable learning rate compared to the full-rank baseline, which usually uses a learning rate $\leq 0.001$ to avoid spikes in the training loss."

Learning rate. The paper uses a learning rate of $0.01$ for GaLore across all model sizes (60M to 7B), with the scale factor $\alpha = 0.25$ making the effective learning rate $0.0025$. Full-rank training uses smaller learning rates ($\leq 0.001$) to avoid loss spikes. The paper finds GaLore "tends to be stable with the same learning rate across different model sizes" (Appendix C.1), suggesting the low-rank projection acts as an implicit regularizer that prevents divergence. Learning rate warmup is applied for the first 10% of training steps, followed by cosine annealing to 10% of the initial learning rate.

4. Key Insights and Innovations

Innovation 1: Shifting the Low-Rank Constraint from Weights to Gradients — A Fundamentally Different Hypothesis About Where Compression Is Safe

The most consequential conceptual move in this paper is not the specific projection algorithm, but the reframing of what should be compressed during training. The dominant paradigm, established by LoRA (Hu et al., 2022) and its many variants, operates on a simple hypothesis: weight updates are low-rank, so constrain the weight matrix to a low-rank parameterization. This hypothesis is convenient — it directly reduces the number of trainable parameters and thus optimizer states — but the paper demonstrates it is also wrong in general for pre-training. The optimal weight matrix for a complex task may require full rank, and forcing low-rank structure on the weights themselves fundamentally caps what the model can learn (Section 3.2: "if the optimal $W^*$ is high-rank, then imposing a low-rank assumption on $W$ never leads to the optimal solution"). The empirical evidence in Table 2 confirms this: LoRA and ReLoRA catastrophically diverge from full-rank training as model scale increases, with LoRA reaching 19.21 perplexity vs. full-rank's 15.56 at 1B parameters — a gap that represents a fundamentally different quality of model, not a minor degradation.

GaLore's alternative hypothesis is that gradients — not weights — become low-rank during training, even when the weights themselves are not. This is a far more subtle claim because it depends on training dynamics rather than static structure. The gradient at any single step might be full-rank, but the trajectory of gradients over time concentrates in a low-dimensional subspace determined by the interaction between the network architecture, the data distribution, and the current optimization landscape. The paper proves this for reversible networks (Theorem 3.2, Lemma 3.3) by showing that the gradient takes the parametric form $G_t = \frac{1}{N}\sum_i (A_i - B_i W_t C_i)$ with PSD $B_i$ and $C_i$, and that under SGD this form forces the stable rank of $G_t$ to converge exponentially to a low value. The crucial insight is that the PSD structure of $B_i$ and $C_i$ — which emerges naturally from the chain rule applied to reversible architectures — creates an eigen-dynamics where gradient components corresponding to larger eigenvalues of the Kronecker-sum matrix $S = \frac{1}{N}\sum_i C_i \otimes B_i$ are exponentially suppressed, leaving only the low-dimensional minimal eigenspace.

Why this shift matters beyond the algorithm. This reframing changes the optimization problem from "how do we train with low-rank weights?" (which LoRA attempts and the paper shows is fundamentally limited) to "how do we identify and track the low-rank subspace that gradients occupy at each phase of training?" (which GaLore solves with periodic SVD). The former is a static constraint on model capacity; the latter is a dynamic compression of optimization signals that preserves full model capacity. This distinction explains why GaLore can match full-rank training perplexity (15.64 vs. 15.56 at 1B) while LoRA cannot — GaLore's weight matrix $W_T = W_0 + \sum_t \tilde{G}_t$ accumulates updates from many different low-rank subspaces over time (Section 4.1), allowing it to become full-rank, while LoRA's $W_T = W_0 + BA$ is structurally constrained to rank at most $r$.

A diagnostic contribution, not just an algorithmic one. The paper's theoretical analysis of why gradients become low-rank provides a diagnostic tool for the field. Prior work had observed gradient low-rankness empirically (Gur-Ari et al., 2018; Zhao et al., 2022; Cosson et al., 2023), but without a mechanistic explanation. The paper's derivation — connecting the gradient structure to the PSD matrices that arise from the network's Jacobian structure, and proving exponential convergence of the stable rank — gives practitioners a way to reason about when gradient compression will work. If a network architecture produces gradients that follow the $A_i - B_i W C_i$ form with PSD $B_i$ and $C_i$, gradient low-rankness is guaranteed; if it does not, the method may fail. This is a testable, falsifiable claim that goes beyond "it works empirically on these benchmarks."

This innovation is fundamental rather than incremental because it challenges the core assumption underlying an entire research lineage (LoRA and its descendants). It does not improve LoRA — it argues that LoRA's premise is wrong for pre-training and proposes a different premise that happens to work better. The shift from "weights are low-rank" to "gradients become low-rank" is a conceptual reframing comparable to the shift from "train a single large model" to "train an ensemble of weak learners" that AdaBoost introduced — it changes what aspect of the learning process we think can be safely compressed.

---

Innovation 2: Multi-Subspace Composition as a Mechanism for Full-Rank Learning from Low-Rank Updates

If GaLore's first innovation is recognizing that gradients are low-rank, the second is recognizing that a single low-rank subspace is insufficient for full-parameter training. This is not obvious from the theory — Theorem 3.8 proves convergence with fixed projections, which might suggest that one good subspace is enough. But convergence in a fixed subspace means convergence to the best solution within that subspace, not the best solution overall. The paper identifies this gap explicitly (Section 4.1) and proposes a mechanism to bridge it: periodically recompute the subspace to track the changing gradient structure, allowing the weight matrix to accumulate updates from many different low-rank subspaces.

This idea — that the composition of low-rank updates across time yields a high-rank final weight — is the mechanism that distinguishes GaLore from both LoRA (which constrains the total weight change to low rank) and ReLoRA (which merges low-rank adaptors periodically but resets optimizer states, losing momentum information). GaLore's multi-subspace composition preserves two things that ReLoRA discards: optimizer state continuity (Adam's $M$ and $V$ accumulate across subspace changes, even though the projection changes, because the compact representation $R_t = P_t^\top G_t$ is recomputed each step) and full-rank weight accumulation (unlike LoRA where $BA$ is always rank-$r$, GaLore's $\sum_i \Delta W_{T_i}$ is a sum of many rank-$r$ matrices that can span the full $m \times n$ space).

The conceptual significance of the subspace change frequency $T$. The introduction of $T$ as a hyperparameter (Section 4.4) is not just an engineering detail — it encapsulates a fundamental tradeoff in optimization under compression. Too-small $T$ means the optimizer's accumulated statistics (momentum, variance) are frequently disrupted by projection changes, violating the constant-projection assumption that Theorem 3.8 requires for guaranteed convergence. Too-large $T$ means the optimization wastes iterations in a subspace that is no longer the most productive direction for loss reduction. The paper's finding that a broad range of $T$ (50–1000) works well (Figure 5, left) is itself informative: it suggests that gradient subspaces change slowly enough that the optimizer can make meaningful progress before needing to switch, but not so slowly that the problem reduces to fixed-subspace training. This "sweet spot" is an empirical discovery about neural network training dynamics, not a theoretical guarantee — and it provides practical guidance for applying similar methods to other architectures.

Why this matters beyond the 4× memory savings. The multi-subspace composition idea suggests a general principle for memory-efficient training: track the optimizer's state in a compressed dynamic subspace that adapts to the changing loss landscape, rather than in the full parameter space or in a static low-rank parameterization. This principle could apply to any optimizer that maintains per-parameter state (Adam, Adafactor, Lion) and any architecture where gradients exhibit temporal coherence in their principal subspaces. The paper demonstrates this generality by showing GaLore works with Adam, 8-bit Adam, and Adafactor (Figure 3) — the projection mechanism is optimizer-agnostic because it operates on the gradient before the optimizer sees it.

This innovation is fundamental in its implications but incremental in its mechanism — the mechanism (periodic SVD recomputation) is straightforward, but the recognition that this enables full-rank learning from low-rank updates is a conceptual advance that distinguishes GaLore from all prior low-rank training methods.

---

Innovation 3: The Empirical Demonstrations That Gradient Compression Scales to Practical LLM Pre-Training — And Where It Does Not

Beyond the theoretical reframing, the paper provides two empirical findings that shift the conversation about what is possible with memory-efficient training. These are not algorithmic innovations but feasibility demonstrations that establish new boundaries for the field.

First, the demonstration that a 7B model can be pre-trained on a single consumer GPU (22 GB on an RTX 4090) without model parallelism, checkpointing, or offloading (Section 5.5, Figure 4). This is not just a "look what fits" result — it redefines what "accessible" means for LLM pre-training. Prior to this work, training a 7B model required at least 58 GB of memory (Figure 1), which meant either expensive datacenter GPUs (A100 80GB) or complex distributed training setups. GaLore + 8-bit Adam + per-layer weight updates reduces this to 22 GB — within the 24 GB budget of a consumer GPU. The economics shift dramatically: a single RTX 4090 system (~$3,000 total) can now do what previously required four A100s (~$40,000+ in GPU cost alone) or a multi-GPU server. This is not an algorithmic improvement — it is a practical accessibility breakthrough enabled by the algorithmic improvements.

The paper is transparent about the tradeoffs: the per-GPU batch size is small (up to 500 tokens without activation checkpointing, up to 4096 with checkpointing), and the current implementation has ~17% throughput overhead compared to 8-bit Adam (Table 11). But the paper explicitly notes that small per-GPU batch sizes can be scaled with data parallelism, which requires low inter-GPU bandwidth compared to model parallelism — meaning GaLore enables elastic training across low-bandwidth consumer hardware. This is a specific, testable claim about distributed training that the paper does not fully validate but provides the foundation for.

Second, the finding that LoRA and ReLoRA catastrophically fail at larger model scales for pre-training (Table 2). The gap between full-rank training and LoRA grows from 0.93 perplexity at 60M parameters (34.99 vs. 34.06) to 3.65 at 1B parameters (19.21 vs. 15.56). ReLoRA without full-rank warmup does better (18.33 at 1B) but still lags far behind full-rank (15.56) and GaLore (15.64). This is a negative result that is as important as the positive GaLore results: it establishes that low-rank weight constraints are fundamentally incompatible with large-scale pre-training, not just suboptimal. The paper does not merely claim GaLore is better — it provides evidence that the dominant approach (LoRA) hits a wall that GaLore circumvents. This negative result reframes the research agenda: memory-efficient pre-training should focus on gradient compression, not weight compression.

The theoretical boundary condition. The paper's theory (Lemma 3.3, Section 3.2) explains why gradient compression works for the architectures and losses studied, but also implicitly defines where it might fail. If the gradient does not follow the $A_i - B_i W_t C_i$ form (e.g., for networks with biases, or with losses that break the PSD property of $B_i$ and $C_i$), the low-rank guarantee does not hold. The paper does not explore these boundaries empirically (all experiments use bias-free LLaMA architectures with standard losses), but the theoretical framework provides a diagnostic: measure the stable rank of $G_t$ during training; if it remains high, GaLore's premise is violated and performance will degrade. This is a testable prediction that future work can validate or falsify.

These empirical findings are incremental in nature but fundamental in impact — they do not propose new algorithms but validate that existing algorithmic ideas (gradient projection, multi-subspace composition) scale to practical LLM training regimes, and they establish negative results that close off a major competing approach (static low-rank weights).

---

Innovation 4: Identifying Verifier-Independent Memory Reduction as the Central Bottleneck for Democratized Training — And Proposing a Unified Solution

While the paper's primary narrative is about gradient compression, a broader insight emerges when considering GaLore's position in the ecosystem of memory-efficient training techniques. The paper identifies (implicitly) that optimizer state memory — not weight memory or activation memory — is the dominant bottleneck for making LLM training accessible, and that existing solutions either address a different bottleneck (activation checkpointing reduces activation memory, offloading moves tensors but does not reduce total memory) or sacrifice model quality (LoRA constrains parameters). GaLore is the first method that directly targets optimizer state memory while preserving full-parameter learning.

Why optimizer states are the real bottleneck. For a 7B model, parameters use ~14 GB, gradients use ~14 GB, and Adam states (first and second moments) use ~28 GB — double the parameters. Even if activation memory is eliminated through checkpointing, the 56 GB for parameters + gradients + optimizer states still exceeds consumer GPU capacity. Reduction in optimizer states is therefore the highest-leverage target: every byte saved there is roughly twice as impactful as saving a byte of activation memory (because it enables larger models to fit at all, not just larger batch sizes). GaLore reduces optimizer state memory by 65.5% in BF16 and 82.5% in 8-bit mode (Section 5.5), directly attacking the largest single memory consumer.

GaLore as a composable primitive. The paper's demonstration that GaLore works with 8-bit Adam, per-layer weight updates, and activation checkpointing (Section 5.5) positions it as a composable primitive in a larger memory-efficient training stack, not a standalone solution. The total memory reduction from BF16 Adam baseline to 8-bit GaLore with per-layer updates is 37.92 GB (63.3% total memory reduction for the 7B model). This composability is important because it means GaLore's benefits stack with future improvements in other components — a better activation compression method or a 4-bit optimizer would further reduce total memory without requiring GaLore modifications.

The "unified solution" claim. The paper demonstrates GaLore on both pre-training (LLaMA, up to 7B) and fine-tuning (RoBERTa on GLUE, Gemma-2B and Phi-2 on OASST and Belle), with comparable or better performance than full-rank training and LoRA respectively (Tables 2, 3, 4, 9, 10). This is significant because fine-tuning and pre-training are typically treated as separate problems with separate solutions — LoRA for fine-tuning, full training for pre-training. GaLore's ability to handle both with the same mechanism suggests that gradient low-rankness is a universal property of neural network training, not specific to either regime. The fine-tuning results (Table 4: GaLore rank-4 achieves 85.89 average GLUE score vs. LoRA's 85.61) show that gradient projection does not harm fine-tuning performance, even when LoRA's weight-constraint approach is specifically designed for that setting.

This innovation is conceptual rather than technical — the insight is not in the projection mechanism itself but in recognizing optimizer state memory as the highest-leverage target and gradient projection as a composable, regime-agnostic solution. This framing influences how future research should approach memory efficiency: not as a collection of disconnected tricks (LoRA for fine-tuning, offloading for pre-training, checkpointing for activations), but as a unified stack where gradient compression is the foundational layer.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All pre-training experiments use the C4 dataset (Raffel et al., 2020), a colossal, cleaned version of Common Crawl's web crawl corpus intended for pre-training language models. The paper trains without data repetition across model sizes from 60M to 7B parameters, with total training tokens ranging from 1.1B (60M model) to 19.7B (7B model) as specified in Table 5 (Appendix C.1). Fine-tuning experiments use the GLUE benchmark (Wang et al., 2019) for RoBERTa-Base, with additional fine-tuning evaluations on SQuAD (Rajpurkar et al., 2016), OpenAssistant Conversations (Köpf et al., 2024), and Belle-1M (BELLEGroup, 2023) datasets using Gemma-2B, Phi-2, and LLaMA-7B.

• Base model(s). The paper uses LLaMA-based architectures (Touvron et al., 2023) with RMSNorm and SwiGLU activations (Zhang & Sennrich, 2019; Shazeer, 2020), evaluated at five scales: 60M, 130M, 350M, 1B, and 7B parameters (Table 5 provides architecture details: hidden dimensions, intermediate sizes, heads, and layers per scale). For fine-tuning, the paper uses pre-trained RoBERTa-Base (for GLUE), BERT-Base (for SQuAD), and Gemma-2B, Phi-2, LLaMA-7B (for OASST and Belle). The LLaMA architecture is chosen because it represents "contemporary LLMs" and the memory bottleneck at 7B scale (58 GB minimum for full-rank training) makes memory efficiency practically critical.

• Metrics. Pre-training performance is measured by validation perplexity on held-out C4 data, reported at multiple training step intervals (e.g., 40K, 80K, 120K, 150K steps for the 7B model). Fine-tuning performance is measured by task-specific metrics on GLUE (Matthew's correlation for CoLA, Pearson correlation for STS-B, F1 for MRPC/QQP, accuracy for others) and average score across all GLUE tasks. For SQuAD, both Exact Match and F1 scores are reported. For OASST and Belle, testing perplexity is reported. Memory usage is measured in gigabytes (GB) of GPU memory at a token batch size of 256 (Figure 4, Table 11), with breakdowns for optimizer states, weight gradients, parameters, and activations. Throughput is measured in tokens per second (Table 11).

• Baselines. The paper compares against five distinct methods:

  • Full-Rank: Standard Adam optimizer with full-rank weights and optimizer states (the quality ceiling).
  • Low-Rank: Traditional low-rank weight factorization $W = BA$ where both $B$ and $A$ are learned from scratch (Kamalakara et al., 2022).
  • LoRA (Hu et al., 2022): Low-rank adaptors $W = W_0 + BA$ where $W_0$ is the fixed initialization. Uses LoRA alpha of 32 and dropout of 0.05 as default settings.
  • ReLoRA (Lialin et al., 2024): LoRA variant that periodically merges $BA$ into $W$ and resets optimizer states and learning rate. Evaluated without full-rank training warmup for fair comparison.
  • 8-bit Adam (Dettmers et al., 2022): For the 7B experiments, 8-bit Adam serves as the memory-efficient baseline against 8-bit GaLore.

• Generation budget / compute accounting. All methods are compared at equal training steps and data budgets. For pre-training, models consume between 1.1B tokens (60M model, 10K steps) and 19.7B tokens (7B model, 150K steps) with a fixed batch size of 131K tokens and max sequence length of 256 (Table 5). Memory estimates in Table 2 are theoretical, based on the number of BF16 parameters and optimizer state entries for each method. Actual memory measurements in Figure 4 and Table 11 are empirical, measured on a single GPU at token batch size 256 without activation checkpointing or memory offloading (unless explicitly noted).

• Cross-validation / statistical protocol. For pre-training, the paper reports results from single trials due to computational constraints, except for the 7B experiments where only one trial is run without hyperparameter tuning. For fine-tuning on GLUE, hyperparameters (learning rate, scale factor $\alpha$) are tuned per task as detailed in Table 7 (Appendix D.1). The paper does not use cross-validation or report confidence intervals. For the 60M–1B experiments, learning rates are tuned from a set of $\{0.01, 0.005, 0.001, 0.0005, 0.0001\}$ and the best is selected based on validation perplexity (Appendix C.1). GaLore uses fixed hyperparameters across all model sizes: learning rate 0.01, scale factor $\alpha = 0.25$, subspace change frequency $T = 200$.

Main Quantitative Results

Pre-Training Comparison Against Low-Rank Methods (Table 2, Figure 6)

Headline result. GaLore matches full-rank training perplexity across all model scales (60M to 1B) while using substantially less memory for optimizer states, and dramatically outperforms all low-rank weight methods (Low-Rank, LoRA, ReLoRA) which degrade severely at larger scales.

At 60M parameters (1.1B training tokens): Full-Rank achieves 34.06 perplexity with 0.36G memory for parameters + optimizer states. GaLore achieves 34.88 perplexity with 0.24G memory — a slight perplexity degradation (0.82) for a 33% memory reduction. LoRA achieves 34.99 perplexity (comparable to GaLore at this small scale) but uses 0.36G memory. ReLoRA achieves 37.04, already showing degradation. Low-Rank (factorized weights) performs catastrophically: 78.18 perplexity, confirming that training with low-rank weights from scratch is not viable.

At 130M parameters (2.2B tokens): Full-Rank: 25.08 perplexity (0.76G). GaLore: 25.36 perplexity (0.52G) — gap of 0.28, 32% memory reduction. LoRA degrades to 33.92 perplexity (0.80G) — the gap vs. full-rank widens from 0.93 to 8.84. ReLoRA: 29.37 perplexity (0.80G), better than LoRA but still 4.29 worse than full-rank. Low-Rank: 45.51.

At 350M parameters (6.4B tokens): Full-Rank: 18.80 perplexity (2.06G). GaLore: 18.95 perplexity (1.22G) — gap of only 0.15, 41% memory reduction. The GaLore curve closely tracks full-rank throughout training (Figure 6). LoRA: 25.58 (gap of 6.78). ReLoRA: 29.08 (worse than LoRA at this scale — 10.28 behind full-rank). Low-Rank: 37.41.

At 1B parameters (13.1B tokens): Full-Rank: 15.56 perplexity (7.80G). GaLore: 15.64 perplexity (4.38G) — gap of only 0.08, 44% memory reduction. This is the critical result: GaLore is essentially indistinguishable from full-rank training at 1B scale, while using 44% less memory for parameters + optimizer states. LoRA: 19.21 (gap of 3.65, but using 6.17G memory — more than GaLore for worse results). ReLoRA: 18.33 (gap of 2.77, better than LoRA but still far behind). Low-Rank: 142.53 (completely failed).

The scaling trend in the gap between GaLore and LoRA/ReLoRA is the paper's strongest empirical argument. At 60M, LoRA and GaLore are comparable (34.99 vs. 34.88). At 130M, LoRA falls behind by 8.56 perplexity points. At 350M, by 6.63. At 1B, by 3.57. While the absolute gap fluctuates, LoRA never recovers to GaLore's performance, and the trend clearly shows that static low-rank weight constraints become increasingly incompatible with pre-training as model scale grows. ReLoRA does better than LoRA at 1B (18.33 vs. 19.21) but still lags GaLore by 2.69 perplexity points.

Memory analysis (Table 2, Table 6). The theoretical memory estimates in Table 2 include both weight parameters and optimizer states. Table 6 breaks these apart. For the 1B model: GaLore uses 2.60G for weights (same as full-rank) and 1.78G for optimizer states (vs. 5.20G for full-rank — a 66% reduction). LoRA uses 3.79G for weights (1.19G more than full-rank because it must store both the frozen $W_0$ and the adaptor $BA$) and 2.38G for optimizer states. Despite having fewer trainable parameters, LoRA uses more total memory than GaLore at 1B (6.17G vs. 4.38G) because the adaptor parameters still require optimizer states and the frozen weights must be retained.

Figure 6 (training progression). The training curves show that GaLore not only matches final perplexity but closely follows the full-rank training trajectory throughout training, and even "converges slightly faster at the beginning of the training" (Appendix C.3). LoRA's training curves (where shown) diverge from full-rank early and never recover.

Integration with Memory-Efficient Optimizers (Figure 3)

Headline result. GaLore composes with various optimizers without degrading convergence, and 8-bit GaLore (GaLore + 8-bit Adam) provides the best memory-performance tradeoff.

Figure 3 evaluates GaLore applied to three optimizers — AdamW, 8-bit Adam, and Adafactor — for pre-training LLaMA 1B on C4 dataset for 10K steps. The validation perplexity curves show that applying GaLore (rank 512 or 1024) to each optimizer does not significantly affect convergence: the GaLore-augmented curve tracks the baseline optimizer curve closely. Specifically:

• AdamW + GaLore (rank=1024): Tracks the full AdamW baseline almost exactly.
• AdamW + GaLore (rank=512): Slightly higher perplexity but still close.
• 8-bit Adam + GaLore (rank=512, 1024): Similar convergence to 8-bit Adam baseline, with the memory footprint reduced by "up to 62.5% on top of the memory savings from using 8-bit Adam."
• Adafactor + GaLore (rank=512, 1024): GaLore applies successfully to Adafactor, which already uses factorized second-moment estimates, achieving further memory reduction.

Since 8-bit Adam already requires less memory than AdamW or Adafactor, the paper designates 8-bit GaLore (GaLore with 8-bit Adam) as the default configuration for the 7B experiments. This is a practical engineering choice: 8-bit GaLore provides the deepest memory savings while matching convergence, making it the best candidate for fitting large models on consumer hardware.

Scaling to LLaMA 7B (Table 3)

Headline result. 8-bit GaLore achieves comparable perplexity to 8-bit Adam when pre-training LLaMA 7B on 19.7B tokens, while using 18 GB vs. 26 GB of memory, confirming that gradient compression scales to practical LLM training regimes.

Table 3 reports perplexity at four checkpoints (40K, 80K, 120K, 150K steps) for LLaMA 7B trained with 8-node parallelism on 64 A100 GPUs. Due to computational constraints, only a single trial is run without hyperparameter tuning:

The final perplexity gap between 8-bit GaLore (14.65) and 8-bit Adam (14.61) is 0.04 — negligible. Notably, 8-bit GaLore starts slightly better (17.94 vs. 18.09 at 40K steps) and maintains rough parity throughout, suggesting that the projection does not introduce systematic bias that compounds over training. The memory savings are substantial: 18 GB vs. 26 GB represents a 30.8% reduction in total training memory at 7B scale.

Important caveats. This is a single-trial result without hyperparameter tuning for either method. The 8-bit Adam baseline is not the same as full-rank BF16 Adam — it already incorporates memory reduction through quantization. The comparison is therefore between two memory-efficient methods (both using 8-bit states), not between GaLore and the strongest possible full-rank baseline. The paper does not report a BF16 full-rank 7B result due to memory constraints, which is precisely the problem GaLore solves — but it means the "comparable performance" claim is relative to a quantized baseline, not to full-precision training.

Memory and Throughput Measurements (Figure 4, Table 11, Figure 1/2 in main text)

Headline result. 8-bit GaLore with per-layer weight updates reduces total training memory for LLaMA 7B by 63.3% compared to BF16 Adam baseline (from ~60 GB to ~22 GB), enabling training on a single consumer GPU with 24 GB VRAM.

Figure 4 shows memory usage across model sizes (60M, 130M, 350M, 1B, 7B) for four configurations: BF16 Adam baseline, BF16 GaLore, 8-bit Adam, 8-bit GaLore, and 8-bit GaLore (retaining grad, which disables per-layer weight updates). The key numbers at 7B scale:

• BF16 Adam: ~60 GB (exceeds any single GPU)
• 8-bit Adam: ~47 GB (exceeds consumer GPUs)
• 8-bit GaLore: ~28 GB (close to 24 GB limit)
• 8-bit GaLore (layer-wise): ~22 GB (fits in 24 GB)

The memory breakdown (described in Section 5.5 text; the actual breakdown figure is referenced as "Fig. 2" but this refers to the main paper's memory breakdown visualization, not Figure 2 of the paper which shows subspace composition):

• Optimizer states: 8-bit GaLore reduces these from 14.6 GB (8-bit Adam) to 5.0 GB — a 65.5% reduction, consistent with the theoretical projection from full $mn$ entries to $(m+n)r$ entries.
• Weight gradients: Per-layer weight updates eliminate the storage of full weight gradients, saving 13.5 GB.
• Parameters: 14.0 GB (unchanged — GaLore learns full parameters).
• Activations: 2.0 GB.

The total reduction from BF16 Adam baseline (~60 GB) to 8-bit GaLore with per-layer updates (~22 GB) is 37.92 GB or 63.3%.

Table 11 measures throughput for LLaMA 1B (not 7B, due to the complexity of benchmarking at 7B scale). With per-layer updates enabled:

• 8-bit Adam: 1205 tokens/second (6.07 samples/second at batch size 256)
• 8-bit GaLore: 1019 tokens/second (5.13 samples/second)
• Overhead: ~15.5% throughput reduction

Without per-layer updates (retaining gradients for all layers before updating):

• 8-bit Adam: 1570 tokens/second (7.90 samples/second)
• 8-bit GaLore (no per-layer): 1109 tokens/second (5.59 samples/second)
• Overhead: ~29.4% throughput reduction

The throughput analysis reveals an important design tradeoff. Per-layer weight updates reduce peak memory (by eliminating gradient storage) but lower throughput by ~23% for 8-bit Adam and ~8% for 8-bit GaLore (comparing with vs. without per-layer updates). Disabling per-layer updates for GaLore improves throughput by 8.8% (from 1019 to 1109 tokens/second) but increases memory. For the 7B model in Figure 4, the memory gap between "8-bit GaLore" and "8-bit GaLore (retaining grad)" suggests that retaining gradients would push the 7B model slightly over the 24 GB budget. This means the per-layer update strategy is essential for consumer GPU training, and the 15.5% throughput overhead is the price for fitting in 24 GB.

The paper explicitly notes that their throughput results "do not require offloading strategies or checkpointing, which can significantly impact training throughput" — meaning the 1019 tokens/second figure for 8-bit GaLore is not directly comparable to systems that use activation checkpointing (which trades throughput for memory) or CPU offloading (which adds PCIe latency). This is a strength of the method (no need for these techniques) but makes throughput comparisons to other memory-efficient systems potentially misleading without controlling for these techniques.

Memory-Efficient Fine-Tuning (Tables 4, 8, 9, 10)

Headline result. GaLore matches or exceeds LoRA for fine-tuning on GLUE, SQuAD, OASST, and Belle benchmarks while using less memory, demonstrating that gradient projection works for fine-tuning as well as pre-training.

GLUE benchmark (Table 4). Fine-tuning RoBERTa-Base on GLUE tasks:

At rank 4, GaLore achieves an average score of 85.89 vs. LoRA's 85.61 — a 0.28 improvement — while using 4 MB less memory (253M vs. 257M parameters). At rank 8, the scores are nearly identical (85.94 for GaLore vs. 85.93 for LoRA), with GaLore using 7 MB less memory. GaLore outperforms LoRA on MRPC (both ranks), RTE (both ranks), SST2 (both ranks), MNLI (rank 8), and QNLI (rank 4); LoRA leads on CoLA (both ranks), STS-B (rank 8), and QQP (rank 8).

The memory advantage, while small in absolute terms for fine-tuning, is structurally significant. GaLore never needs to store a separate low-rank adaptor $BA$ alongside the frozen weights $W_0$ — the full weight $W$ is updated directly. In LoRA, $W_0$ is frozen and $BA$ is learned, requiring storage for both. In GaLore for fine-tuning, the same algorithm is used as in pre-training: the weight $W$ is the only parameter storage, and the projection matrices $P$ (computed on-the-fly from the gradient SVD) are the only additional memory cost. This means GaLore's fine-tuning memory is simply the model parameters plus compact optimizer states — no adaptor overhead.

SQuAD (Table 8). Fine-tuning BERT-Base with rank 16: GaLore achieves 80.52 Exact Match and 88.29 F1, outperforming LoRA (77.99 EM, 86.11 F1) by substantial margins (2.53 EM, 2.18 F1), and closely approaching the full fine-tuning baseline (80.83 EM, 88.41 F1). This suggests that for extractive QA, LoRA's low-rank weight constraint is more limiting than GaLore's gradient projection — consistent with the pre-training finding that gradient compression preserves more model capacity.

OASST and Belle (Tables 9, 10). Fine-tuning on conversational and instruction-following datasets across three model families (Gemma-2B, Phi-2, LLaMA-7B) at rank 128:

• OASST (Table 9): GaLore achieves testing perplexity of 4.51 (Gemma), 3.83 (Phi), 2.95 (LLaMA) vs. LoRA's 4.56, 4.24, 2.94. GaLore matches or slightly outperforms LoRA on all three models, with the most dramatic gap on Phi-2 (3.83 vs. 4.24).
• Belle (Table 10): GaLore achieves 5.35 (Gemma), 2.62 (Phi), 2.28 (LLaMA) vs. LoRA's 5.37, 2.75, 2.30. Again, GaLore matches or outperforms, with the largest gap on Phi-2 (2.62 vs. 2.75).

The consistency across model families (Gemma, Phi, LLaMA) and datasets (GLUE, SQuAD, OASST, Belle) suggests that GaLore's gradient projection approach is not architecture-specific or task-specific — it generalizes across the fine-tuning landscape. This supports the paper's claim that GaLore provides a "full-stack memory-efficient training strategy for both LLM pre-training and fine-tuning" (Section 5.4).

Ablation Studies and Robustness Checks

Subspace change frequency $T$ (Figure 5, left): On a 130M model, varying $T$ from 50 to 1000 steps shows that both "too frequent and too slow changes of subspaces hurt the convergence." The paper reports that values between $T = 50$ and $T = 1000$ make "no much difference," and selects $T = 200$ as the default. The qualitative trend is that for small rank $r$, more frequent switching is beneficial (to avoid wasting optimization steps in a stale subspace), while for large $r$, the gradient updates naturally cover more subspaces, providing cushion against stale projections. This finding is reported qualitatively in the text but the actual perplexity curves for different $T$ are shown in Figure 5 (left).

Rank $r$ and training duration (Figure 5, right): On a 130M model, training with rank 128 for 80K steps achieves lower loss than training with rank 512 for 20K steps. This demonstrates a memory-compute tradeoff: in severely memory-constrained settings, one can reduce the rank to stay within budget and train for more steps to preserve performance. The paper interprets this as GaLore enabling practitioners to "trade-off between memory and computational cost." The convergence slowdown is described as "nearly linear" with decreasing rank within a certain range — a claim that is supported qualitatively by the figure but not quantified with convergence rate measurements.

Optimizer compatibility (Figure 3, discussed above): GaLore is tested with AdamW, 8-bit Adam, and Adafactor, showing that the projection mechanism does not disrupt convergence for any of these optimizers. This is an important robustness check because it demonstrates that GaLore operates independently of the specific gradient processing in the optimizer — it compresses the gradient before the optimizer sees it, making it compatible with any optimizer that accepts a gradient tensor as input.

Fine-tuning rank and scale factor (Tables 7, 4): For GLUE fine-tuning, the paper sweeps over rank ($r = 4, 8$) and tunes the scale factor $\alpha$ per task (e.g., $\alpha = 4$ for rank-4 MNLI but $\alpha = 2$ for rank-8 MNLI and SST-2). The results in Table 4 show that GaLore is not overly sensitive to these choices — both rank 4 and rank 8 produce similar average scores (85.89 vs. 85.94) that outperform LoRA at the corresponding ranks. The per-task tuning in Table 7 shows the learning rate and $\alpha$ choices, though the paper does not report an ablation over $\alpha$ to quantify sensitivity.

Single-projection design choice (Algorithm 2, Table 1): The paper uses only one projection matrix $P$ (when $m \leq n$) rather than both $P$ and $Q$. This is justified theoretically by the memory analysis in Table 1 and empirically by the results matching full-rank training, but there is no direct ablation comparing single-projection vs. dual-projection GaLore to quantify how much performance (if any) is lost by this design simplification. This is a missing experiment that would strengthen the claim that single-projection suffices.

Per-layer weight updates (Table 11, Figure 4): The comparison between "8-bit GaLore" and "8-bit GaLore (retaining grad)" in Figure 4 shows the memory savings from per-layer updates (~6 GB for the 7B model). Table 11 quantifies the throughput cost: enabling per-layer updates reduces throughput from 1109 to 1019 tokens/second for the 1B model (8.8% overhead). This is a clean ablation demonstrating the memory-throughput tradeoff of this specific technique.

Full-rank warmup for ReLoRA (Table 2): The paper evaluates ReLoRA without full-rank training warmup "for a fair comparison" with GaLore (which also has no warmup phase). This is a deliberate ablation: it tests whether ReLoRA's reported performance depends on the warmup phase. The results (18.33 at 1B vs. GaLore's 15.64) confirm that without warmup, ReLoRA cannot match full-rank training or GaLore. The paper does not evaluate ReLoRA with warmup, which would be a stronger baseline but would compromise the memory-efficiency comparison (since warmup uses full memory).

Gradient low-rankness across training (no explicit figure, discussed in Section 3 and Appendix B): The paper's theoretical claims about gradient low-rankness are supported by the practical success of GaLore rather than by direct measurement of gradient stable rank during training. There is no figure showing the stable rank of $G_t$ over time for LLaMA models, which would directly validate Lemma 3.3 and Corollary 3.4. The theoretical analysis proves low-rankness for reversible networks and Transformer FFN layers (Appendix B.3), but the empirical validation is indirect: GaLore works, therefore gradients must be low-rank. A direct measurement of gradient singular value spectra during training would strengthen the paper's central theoretical claim.

Critical Assessment

Claim 1: "GaLore reduces memory usage by up to 65.5% in optimizer states"

This claim is supported, but "up to" does important work. The 65.5% figure comes from the comparison of 8-bit GaLore vs. 8-bit Adam optimizer states for the 7B model (Section 5.5: 5.0 GB vs. 14.6 GB). This is a specific comparison at a specific scale with a specific optimizer configuration (8-bit). The BF16 GaLore vs. BF16 Adam comparison shows smaller relative savings because the 8-bit quantization already provides a 4× reduction independently of GaLore. Table 2 shows that at 1B scale in BF16, GaLore uses 1.78G for optimizer states vs. 5.20G for full-rank — a 65.8% reduction. So the "up to 65.5%" claim is accurate but context-dependent: it holds when comparing against full-precision (BF16/FP32) optimizer states, or when GaLore's projection savings stack with quantization savings.

The 82.5% figure (8-bit GaLore optimizer memory reduction) is the combination of GaLore projection and 8-bit quantization — the 8-bit quantization already provides ~75% reduction (32-bit to 8-bit), and GaLore reduces the number of entries further. The paper is transparent about this stacking, but the headline figure conflates two independent mechanisms.

Claim 2: "GaLore maintains both efficiency and performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset with up to 19.7B tokens"

Supported for the configurations tested, but with important scope limitations. The 1B results (Table 2) are thorough: GaLore achieves 15.64 vs. full-rank's 15.56, a gap of 0.08, after 13.1B tokens. The 7B results (Table 3) compare GaLore only against 8-bit Adam (14.65 vs. 14.61, gap of 0.04), not against full-precision full-rank training. The "maintains performance" claim at 7B is therefore relative to a quantized baseline, not to the strongest possible training configuration.

The 19.7B token budget is modest for a 7B model. Chinchilla-optimal training would require ~140B tokens for a 7B model. The paper acknowledges this implicitly by reporting results at intermediate checkpoints (Table 3 shows perplexity still improving at 150K steps), but does not discuss whether GaLore's performance relative to baselines changes with longer training. The theoretical analysis (Lemma 3.3) predicts that gradients become more low-rank over time, which would make GaLore more effective at longer training horizons — but this is untested.

Claim 3: "We demonstrate, for the first time, the feasibility of pre-training a 7B model on consumer GPUs with 24GB memory without model parallel, checkpointing, or offloading strategies"

This claim is supported but with important caveats about batch size and throughput. The 22 GB memory measurement (Section 5.5) for LLaMA 7B with 8-bit GaLore and per-layer weight updates is an empirical measurement on a single GPU with a token batch size of 256. This fits within 24 GB. However:

• The per-GPU batch size is small (256 tokens, or ~1 sample of max sequence length 256). The paper acknowledges this and argues it can be scaled with data parallelism, which requires lower inter-GPU bandwidth than model parallelism. This is true, but the convergence behavior of large-batch training has well-known degradation effects that are not studied in this paper.
• Activation checkpointing can increase the batch size to 4096 tokens (as noted in Section 5.5), but activation checkpointing was explicitly listed as something GaLore avoids. Using it changes the claim from "without checkpointing" to "with minimal checkpointing."
• Throughput is low. At 1019 tokens/second for 1B (Table 11), training a 7B model on a single consumer GPU would take impractically long for full pre-training. The paper does not report 7B throughput, but the 1B numbers suggest that while feasibility is demonstrated, practicality for full pre-training from scratch on a single consumer GPU remains limited without distributed data parallelism.
• The "without model parallel" claim is technically true for a single GPU, but the 7B pre-training experiment (Table 3) was run on 64 A100 GPUs with 8-node parallelism — not on a single RTX 4090. The claim is about feasibility given the memory measurements, not about actually running the full 19.7B token training on a single consumer GPU.

Claim 4: "GaLore outperforms other low-rank methods and achieves comparable performance to full-rank training"

Strongly supported for pre-training (Table 2), mixed for fine-tuning (Table 4). For pre-training, the evidence is clear: GaLore dramatically outperforms LoRA, ReLoRA, and Low-Rank across all scales, and matches full-rank within ~0.1-0.8 perplexity depending on scale. For fine-tuning on GLUE, GaLore's advantage over LoRA is modest (0.28 average score at rank 4, 0.01 at rank 8) and comes with a slight performance gap vs. full fine-tuning (85.89 vs. 86.28 at rank 4). The fine-tuning results support "comparable to LoRA" more strongly than "comparable to full fine-tuning" — the gap to full fine-tuning (0.39 at rank 4, 0.34 at rank 8) is small but consistent.

Missing Experiments That Would Strengthen the Paper

1. Direct measurement of gradient stable rank over time. The paper's central theoretical claim — that gradients become low-rank during training — is validated only indirectly through GaLore's empirical success. A figure showing the singular value spectrum of $G_t$ at different training stages for a LLaMA model would directly test Lemma 3.3. The theoretical analysis predicts exponential decay of stable rank; measuring this decay rate and comparing it to the theoretical bound would be a strong validation.

2. Comparison against GaLore with random projections instead of SVD. Hao et al. (2024) proposed Flora, which uses random projections to compress gradients. Comparing SVD-based GaLore against random-projection GaLore would test whether the careful eigenstructure tracking (Theorem 3.8) matters, or whether any low-rank projection suffices. This would distinguish the theoretical contribution from the empirical observation that gradients are low-rank.

3. Evaluation on a non-LLaMA architecture. All pre-training experiments use LLaMA-based architectures. Testing on a different architecture family (e.g., GPT-style, SSM-based) would test whether the gradient low-rankness property is architecture-dependent. The theory covers reversible networks and Transformer FFN layers, but not attention mechanisms specifically. An architecture where gradients do not follow the $A_i - B_i W C_i$ form might cause GaLore to fail.

4. Longer training horizons. The maximum token budget is 19.7B for 7B, which is well below Chinchilla-optimal. Testing GaLore at 100B+ tokens would reveal whether the multi-subspace composition mechanism successfully tracks gradient subspaces over full training durations, or whether the projection quality degrades over very long horizons.

5. Memory-throughput Pareto frontier. The paper reports throughput only at 1B scale (Table 11) and does not systematically vary rank $r$, subspace frequency $T$, or per-layer update strategy to map the memory-throughput tradeoff space. A Pareto frontier analysis would help practitioners choose optimal configurations for their hardware constraints.

6. Larger-scale fine-tuning. The GLUE, SQuAD, OASST, and Belle fine-tuning experiments use relatively small models (RoBERTa-Base, BERT-Base, 2B-7B models). Testing GaLore for fine-tuning 13B-70B models would validate the "full-stack" claim at scales where memory efficiency is practically necessary.

Conditional Nature of the Claims

The central claim — that GaLore enables full-parameter learning with dramatically reduced memory — holds when:

• The base architecture produces gradients that become low-rank during training (reversible networks, Transformers with the JoMA structure).
• The rank $r$ is chosen appropriately for the model scale (too small and convergence suffers, Figure 5 right).
• The subspace change frequency $T$ is in the broad acceptable range (50–1000).
• Per-layer weight updates are used for maximum memory savings (at ~8% throughput cost).
• 8-bit quantization is combined with GaLore for consumer-GPU deployments.

The claim does not necessarily hold when:

• Training on architectures without the gradient structure that guarantees low-rankness.
• Training on very long horizons (beyond 19.7B tokens) — extrapolation is plausible but untested.
• Using extremely small ranks ($r \ll 128$ for 130M+ models) — convergence may require impractical numbers of steps.

Overall, the experimental evidence strongly supports the paper's core claims about memory reduction and performance preservation at the tested scales and configurations. The 7B consumer GPU feasibility demonstration is the most impactful result, though it comes with practical caveats about throughput and batch size that the paper acknowledges. The weakest empirical link is the absence of direct gradient low-rankness measurements, which leaves the theoretical narrative somewhat decoupled from the experimental validation.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Numbers

The assumption or constraint. The paper's compute-optimal framework depends on knowing each prompt's difficulty before deciding how to allocate the inference budget. The method for estimating difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. The paper explicitly acknowledges this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution. Since 2048 samples exceeds the largest test-time budgets studied (256–512 generations), the difficulty estimation cost could dominate the total compute budget, potentially negating or even reversing the claimed efficiency gains. The 4× figure should therefore be understood as an upper bound on achievable efficiency that assumes free difficulty labels — an assumption that does not hold in deployment.

What evidence exists in the paper. The paper provides no measurement of the total cost including difficulty estimation. The compute-optimal scaling curves in Figures 4 and 8 plot accuracy against strategy execution budget only, not total budget. The paper notes that predicted difficulty bins perform similarly to oracle bins, which eliminates the need for ground-truth labels but does nothing to reduce the 2048-sample estimation cost.

Mitigation status. The paper acknowledges this limitation and suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), as well as adaptive difficulty estimation that amortizes the cost into the problem-solving process itself (Section 3.2). Neither approach is developed or evaluated. Without a cheap difficulty estimator, the compute-optimal framework is not directly deployable as described.

---

The Method Completely Fails on Hard Problems

The assumption or constraint. GaLore's gradient projection mechanism is predicated on the existence of a low-rank structure in the gradient $G_t$ that can be captured by periodic SVD. This structure emerges naturally from the training dynamics of reversible networks and Transformer FFN layers (Lemma 3.3, Appendix B.3), but the theoretical guarantees assume specific architectural properties (bias-free layers, reversible activations, the JoMA framework for FFN layers). The paper does not theoretically analyze attention layers, which are the other major component of Transformers, and the JoMA analysis for FFN layers requires assumptions (fixed embeddings, stationary backpropagated gradients) that may not hold in practice.

The consequence. If the gradient does not become low-rank — either because the architecture violates the reversible network conditions, or because the training phase does not induce the eigen-dynamics that suppress gradient components — then projecting $G_t$ to a rank-$r$ subspace loses information critical for optimization. The optimizer may stall or converge to a worse solution than full-rank training. The paper provides no diagnostic for detecting when gradient low-rankness fails, and no fallback mechanism for switching to full-rank training if needed. For architectures like vision transformers, diffusion models, or models with bias terms, the gradient structure may differ in ways that prevent GaLore from working. The paper explicitly lists applying GaLore to vision transformers and diffusion models as "open problems" (Section 7), implicitly acknowledging this uncertainty.

What evidence exists in the paper. The paper validates GaLore on LLaMA architectures only (Table 2, Table 3, Figure 3). There is no experiment measuring the stable rank of $G_t$ during training to confirm it is actually low — the evidence is entirely indirect (GaLore works, therefore gradients must be low-rank). The theoretical coverage is partial: reversible networks cover linear layers and FFN activations, but the Transformer attention mechanism is not analyzed. The JoMA proof for FFN layers (Lemma B.3) assumes fixed embeddings and stationary gradients, which are approximations that may not hold early in training when representations change rapidly.

Mitigation status. Not addressed. The paper provides no mechanism for detecting when the low-rank gradient assumption fails and no adaptive strategy for falling back to higher-rank or full-rank updates. Section 7 lists applying GaLore to other architectures as future work, implicitly acknowledging that the current validation is narrow.

---

Single Benchmark, Single Model Family — No Evidence of Cross-Architecture or Cross-Domain Generalization

The assumption or constraint. All experiments in the paper use LLaMA-based architectures (RMSNorm, SwiGLU activations, bias-free layers) trained on the C4 dataset for pre-training, and RoBERTa/BERT/Gemma/Phi for fine-tuning. The theoretical analysis covers reversible networks (which exclude bias terms and require specific activation functions) and Transformer FFN layers under the JoMA framework. The paper does not evaluate on non-Transformer architectures (e.g., state-space models, RNNs, convolutional models), on tasks other than language modeling (e.g., code generation, mathematical reasoning), or on data modalities other than text.

The consequence. Several aspects of GaLore's performance could be model- or task-specific. The gradient low-rankness property depends on the network architecture and loss function — Theorem 3.2 proves it for reversible networks with ℓ₂ loss and logsoftmax loss, but does not cover architectures with bias terms, layer normalization (which is present in LLaMA as RMSNorm but not analyzed), or attention mechanisms. If a different architecture produces gradients that do not follow the $A_i - B_i W_t C_i$ form, GaLore's projection could discard essential gradient information. Similarly, the optimal hyperparameters (rank $r$, subspace frequency $T$, scale factor $\alpha$) may not transfer to other architectures or datasets — the paper uses the same values ($T=200$, $\alpha=0.25$) across all LLaMA scales, but there is no evidence these generalize. The paper states that "we believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this belief is not empirically tested.

What evidence exists in the paper. The paper evaluates on five LLaMA scales (60M to 7B) for pre-training, and on RoBERTa, BERT, Gemma-2B, Phi-2, and LLaMA-7B for fine-tuning — all Transformer variants. There are no experiments with non-Transformer architectures, non-text modalities, or non-language-modeling objectives. The fine-tuning results span GLUE, SQuAD, OASST, and Belle, providing some task diversity, but all are NLP tasks with relatively similar structures.

Mitigation status. Not addressed in the current paper. Section 7 lists applying GaLore to "various models such as vision transformers and diffusion models" as future work, explicitly acknowledging the narrow evaluation scope. The theoretical analysis provides a framework for predicting when gradient low-rankness will hold (reversible network architectures), but this framework is not empirically validated beyond Transformer FFN layers.

---

The Throughput Overhead and Latency Implications Are Not Fully Characterized, Especially at Scale

The assumption or constraint. GaLore introduces computational overhead from two sources: (1) periodic SVD computation on the gradient matrix $G_t \in \mathbb{R}^{m \times n}$ every $T$ steps, which costs $O(\min(m,n) \cdot mn)$; (2) per-layer weight updates, which serialize the backward pass and optimizer step, preventing parallel gradient computation across layers. The paper reports throughput at only the 1B scale (Table 11) and does not report overhead for the 7B model. The headline memory numbers (22 GB for 7B training on a consumer GPU) rely on per-layer weight updates being enabled.

The consequence. For practitioners deciding whether to use GaLore, throughput matters as much as memory. The 1B-scale throughput numbers show a 15.5% overhead for 8-bit GaLore vs. 8-bit Adam with per-layer updates (1019 vs. 1205 tokens/second), and a 29.4% overhead without per-layer updates (1109 vs. 1570 tokens/second). If these overheads scale similarly or worsen at 7B, the wall-clock time for training a 7B model on a consumer GPU could be substantially longer than the memory numbers alone suggest. Moreover, the per-layer update strategy introduces serial dependencies that prevent overlapping gradient computation across layers — this is a latency cost not captured by throughput measurements at small scale where compute-bound versus memory-bound dynamics may differ. The paper does not report SVD computation time as a fraction of total step time, making it difficult to assess whether the $T=200$ frequency is truly "negligible" (as claimed) at 7B scale where $m = n = 4096$ and SVD of a $4096 \times 4096$ matrix is non-trivial.

What evidence exists in the paper. Table 11 provides the only throughput measurements, and only for LLaMA 1B. The paper states that SVD overhead is "< 10% compared to other memory-efficient training techniques such as memory offloading" (Section 4.1), but this comparison is against offloading overhead (which involves PCIe transfers), not against the baseline optimizer's step time. There is no throughput measurement at 7B scale, no latency analysis of the SVD operation as a function of matrix size, and no characterization of how the overhead scales with model dimension.

Mitigation status. Partial. The paper acknowledges that "the current implementation of 8-bit GaLore achieves 1019.63 tokens/second, which induces 17% overhead compared to 8-bit Adam implementation" (Appendix E) and notes that "disabling per-layer weight updates for GaLore achieves 1109.38 tokens/second, improving the throughput by 8.8%." The paper also states that "we leave optimizing the efficiency of GaLore implementation for future work" (Section 5.5). However, no analysis of SVD cost scaling, no 7B throughput numbers, and no latency characterization are provided.

---

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate That Is Only Partially Mitigated

The assumption or constraint. The revision model (Section 6.1) is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target. This means the model never sees trajectories where a correct answer should be preserved. At test time, when the model generates a correct answer during the revision chain and then conditions on it to produce the next revision, it may incorrectly "revise" the correct answer into an incorrect one because it has no training signal for what to do when the current answer is already correct. The paper reports that approximately 38% of correct answers get converted back to incorrect ones using a naive approach.

The consequence. This 38% reversion rate means that longer revision chains are not monotonically improving — the model can oscillate between correct and incorrect answers, and simply taking the final answer in a long chain would be worse than stopping earlier. The paper mitigates this with majority voting or verifier-based selection across the entire chain, picking the best answer from any point rather than always taking the last revision. However, this mitigation is imperfect: it requires generating an entire chain and then selecting, which wastes computation on revisions that degrade performance. It also means the revision process is not self-correcting in the sense of reliably improving — it is better described as "generate many candidates, some of which happen to be better, and hope the selector picks the right one." For latency-sensitive applications where generating long chains is expensive, this inefficiency matters.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. Figure 6 (left) shows that pass@1 at each step in the revision chain improves gradually and then plateaus around 24–25% after steps 15–20, rather than continuing to improve — the plateau is consistent with correct answers being revised back to incorrect ones at a rate that balances new correct answers being generated. The paper does not break down revision trajectories to show how often the model oscillates versus steadily improves.

Mitigation status. Partial. The paper mitigates the reversion problem with within-chain selection (majority voting or verifier-based) rather than always taking the last revision. However, the selection mechanism adds its own cost and is not perfectly reliable — the verifier can make mistakes, and majority voting requires generating enough correct answers to form a majority. A more principled solution, such as training the model with explicit "no revision needed" tokens or training on trajectories that include correct-to-correct transitions, is not explored. The ReST$^{EM}$ experiment (Appendix K, Figure 16) suggests that alternative revision training procedures can make the problem worse: the ReST$^{EM}$-trained model showed degraded performance with sequential revisions, possibly because on-policy data collection amplified spurious correlations. This indicates the reversion problem is sensitive to training methodology in ways that are not fully understood.

---

The Method Has Not Been Validated at Chinchilla-Optimal Training Scales or on Production-Grade Training Runs

The assumption or constraint. The maximum training budget in the paper is 19.7B tokens for the 7B model (Table 3), which is well below the Chinchilla-optimal budget of ~140B tokens for a 7B-parameter model. The longest training run is 150K steps. The paper's theoretical analysis (Lemma 3.3) predicts that gradients become more low-rank over time as training progresses, which would make GaLore more effective at longer horizons — but this prediction is untested. Additionally, the 7B experiment (Table 3) is a single trial without hyperparameter tuning, compared only against 8-bit Adam (not full-precision full-rank training), and run on 64 A100 GPUs with 8-node parallelism — not on a single consumer GPU.

The consequence. The paper's central claim — that GaLore enables full-parameter pre-training with dramatically reduced memory while matching full-rank performance — is validated at moderate scale (up to 1B parameters with rigorous baselines, 7B parameters with a single quantized baseline). It is not validated at the scales and training durations typical of production LLM training. Extrapolating from 1B to 7B, or from 19.7B tokens to 140B+ tokens, involves assumptions about gradient low-rankness, subspace stability, and optimizer dynamics that are plausible but unverified. In particular, the multi-subspace composition mechanism (Section 4.1) must successfully track the evolving gradient structure over potentially hundreds of thousands of steps — if the gradient subspaces drift faster than the recomputation frequency $T$ can track, or if the optimizer states accumulated in one subspace become stale when the projection changes, convergence could degrade at longer horizons.

What evidence exists in the paper. The 7B experiment (Table 3) shows GaLore tracking 8-bit Adam closely over 150K steps, with the perplexity gap stable or narrowing (0.15 at 40K, 0.08 at 80K, 0.12 at 120K, 0.04 at 150K). This suggests the projection is not degrading over this duration, but 150K steps is relatively short for a 7B model. The paper does not report GaLore's performance relative to full-precision full-rank training at any scale above 1B, making it impossible to assess whether the quantization + projection combination introduces a systematic gap that would widen at larger scales or longer horizons. The consumer GPU feasibility demonstration (Section 5.5) is based on memory measurements with a small batch size, not on an actual end-to-end training run on consumer hardware.

Mitigation status. Not addressed. The paper acknowledges computational constraints that prevented longer training runs and more extensive hyperparameter tuning at 7B scale ("Due to computational constraints, we compare 8-bit GaLore with 8-bit Adam with a single trial without tuning the hyperparameters," Section 5.3). The theoretical analysis predicts that gradient low-rankness should improve over time (the exponential decay term in Lemma 3.3), which would make GaLore more robust at longer horizons, but this prediction is not empirically validated. Testing GaLore at Chinchilla-optimal scales is left entirely to future work.

7. Implications and Future Directions

How This Work Changes the Landscape

GaLore fundamentally reframes the memory-efficiency problem in neural network training by shifting the target of compression from weights to gradients. This is not an incremental improvement over LoRA — it is a rejection of LoRA's core premise (that weight updates are low-rank and therefore weights should be parameterized as low-rank) in favor of a more nuanced hypothesis: gradients become low-rank during training even when weights do not, and tracking the evolving gradient subspace enables full-parameter learning with dramatically reduced optimizer memory. The paper proves this hypothesis theoretically for reversible networks and Transformer FFN layers, and validates it empirically at scales up to 7B parameters and 19.7B tokens on LLaMA architectures.

The most immediate landscape change is practical rather than conceptual: GaLore demonstrates that a 7B model can be pre-trained on a single consumer GPU with 24 GB of memory (Figure 4, Section 5.5), a configuration that previously required at least 58 GB and multiple datacenter GPUs. The ~63% reduction in total training memory (from ~60 GB for BF16 Adam to ~22 GB for 8-bit GaLore with per-layer weight updates, Figure 4) lowers the hardware barrier for LLM pre-training from "datacenter cluster" to "single consumer GPU," democratizing access to large-scale model development. This is not a marginal improvement — it is a threshold-crossing result that changes who can participate in pre-training research.

However, the paper also establishes a conditional boundary that previous work had not articulated: GaLore's gradient compression works when the gradient follows the specific parametric form $G_t = \frac{1}{N}\sum_i (A_i - B_i W_t C_i)$ with PSD $B_i$ and $C_i$ (Theorem 3.2, Lemma 3.3). This form emerges naturally from the chain rule applied to reversible networks — which include linear layers without bias, ReLU activations, and ResNet blocks — but does not automatically hold for arbitrary architectures. The paper's theoretical analysis thus serves double duty: it explains why GaLore works on LLaMA-style Transformers, and it provides a diagnostic framework for predicting when it will fail on other architectures. This means the field can now make testable, falsifiable predictions about where gradient compression is applicable, rather than relying on empirical trial-and-error across architectures.

The paper also reconciles a previously confusing picture in the low-rank training literature. Prior work had shown that LoRA works well for fine-tuning (Hu et al., 2022) but fails for pre-training unless combined with full-rank warmup (ReLoRA; Lialin et al., 2024). The paper's results in Table 2 provide a unified explanation: LoRA constrains the total weight change to be low-rank, which is acceptable when starting from a good initialization (fine-tuning) but catastrophically limiting when starting from scratch (pre-training). GaLore constrains only the per-step gradient, accumulating full-rank weights through multi-subspace composition (Section 4.1). This resolution converts a set of apparently contradictory empirical findings — LoRA works for task A but not task B — into a coherent theoretical picture with clear boundary conditions.

A subtle but important shift is that GaLore repositions optimizer state memory as the highest-leverage target for memory reduction. The paper's memory breakdown (Section 5.5) shows that optimizer states account for ~28 GB of the ~60 GB required for LLaMA 7B training — nearly half the total budget. Prior engineering approaches (activation checkpointing, CPU offloading) and prior algorithmic approaches (LoRA) targeted different bottlenecks (activations, weight gradients, or parameter count). GaLore directly attacks the largest single memory consumer while preserving full-parameter learning, establishing optimizer state compression as the most impactful direction for democratized training.

The paper also makes negative results a first-class contribution. Table 2 shows that static low-rank weight methods (LoRA, ReLoRA, Low-Rank) catastrophically degrade at larger scales — LoRA reaches 19.21 perplexity vs. full-rank's 15.56 at 1B parameters, and Low-Rank (factorized weights from scratch) reaches 142.53. These are not marginal degradations; they are complete failures. By publishing these negative results alongside GaLore's positive results, the paper effectively closes off a research direction (static low-rank weight training for pre-training) that had seemed promising, and redirects effort toward gradient compression. This is a landscape-shaping contribution that is rare in the ML literature.

Follow-Up Research This Work Enables

Direct measurement of gradient stable rank during training across architectures. The paper's central theoretical claim — that gradients become low-rank — is validated only indirectly through GaLore's empirical success. The next step is a systematic measurement study: instrument training runs on LLaMA, GPT, SSM, and vision transformer architectures to record the singular value spectrum of $G_t$ for each layer at regular intervals (every 100 steps) over the full training duration. For LLaMA architectures, this would directly test Lemma 3.3's prediction of exponential decay in stable rank; for other architectures, it would establish whether the $A_i - B_i W_t C_i$ gradient form (and its low-rank consequences) generalizes. A strong study would correlate the measured stable rank with GaLore's performance at different projection ranks $r$, establishing an empirical diagnostic ("if stable rank of $G_t$ drops below $X$ by step $Y$, GaLore with rank $Z$ will match full-rank performance") that replaces the current trial-and-error hyperparameter selection.

GaLore with random projections vs. SVD-based projections. The paper uses SVD of $G_t$ to select projection matrices, motivated by Theorem 3.8's requirement that projections capture the top eigenspaces of $B_i$ and $C_i$. Hao et al. (2024) proposed Flora, which uses random projections to compress gradients for fine-tuning. A direct comparison — GaLore with SVD-based projections vs. GaLore with random Gaussian projections (resampled at the same frequency $T$) — would test whether the careful eigenstructure tracking matters, or whether any fixed-dimensional projection suffices. The experiment: pre-train LLaMA 1B with both variants at ranks $r \in \{128, 256, 512\}$, measuring final perplexity and convergence speed. If random projections match SVD projections, the theoretical machinery of Theorem 3.8 is unnecessary for practical performance; if SVD projections significantly outperform, the eigenstructure tracking is a genuine algorithmic contribution rather than merely a theoretical justification. The paper does not run this ablation, and it directly addresses a competing method (Flora) that the authors cite but do not empirically compare against.

Scaling GaLore to Chinchilla-optimal training horizons (~140B tokens for 7B). The paper's longest training run is 19.7B tokens (150K steps) for the 7B model — a fraction of the ~140B tokens that Chinchilla scaling laws prescribe. Lemma 3.3 predicts that gradients become more low-rank over time (the exponential decay term drives stable rank down), which would make GaLore more effective at longer horizons. However, the multi-subspace composition mechanism (Section 4.1) must successfully track the evolving gradient structure across hundreds of thousands of steps — and the optimizer states accumulated in each subspace may become stale when the projection changes. A critical stress test: pre-train LLaMA 7B with 8-bit GaLore for 140B+ tokens and compare against full-precision full-rank training (accepting the cost of running the full-rank baseline on a large cluster for validation purposes). Measure whether the perplexity gap between GaLore and full-rank training widens, narrows, or remains stable over this duration. If GaLore diverges at long horizons (despite the theory predicting the opposite), it would reveal that the multi-subspace switching mechanism has a failure mode not captured by the fixed-projection convergence proof (Theorem 3.8).

GaLore applied to attention layers specifically — theoretical analysis and empirical verification. The paper's theoretical analysis covers reversible networks (linear layers, FFN activations) and Transformer FFN layers (via JoMA, Appendix B.3), but does not analyze attention mechanisms. The attention layer involves the softmax operation and the query-key-value projection structure, which may produce gradients with different spectral properties. A targeted study: isolate attention layers and FFN layers in a LLaMA 1B model, measure the stable rank of $G_t$ for each layer type separately during training, and test whether GaLore with the same rank $r$ works equally well on both. If attention gradients remain higher-rank than FFN gradients throughout training, a hybrid strategy — higher projection rank for attention layers, lower rank for FFN layers — could further reduce memory without degrading performance. This would also provide the theoretical groundwork for extending GaLore to architectures where attention is the dominant component (e.g., vision transformers).

GaLore with learned, low-memory projection matrices. The current method computes projection matrices $P$ via SVD of $G_t$ every $T$ steps, which costs $O(m^2 n)$ for a gradient matrix of size $m \times n$. While the paper reports this overhead as "<10%" (Section 4.1), it is measured against offloading overhead, not against the optimizer step time itself. Moreover, the projection matrices $P$ themselves consume memory ($m \times r$ entries). A natural extension: train a lightweight neural network (a small MLP or a hypernetwork) that takes the layer's input activations and output gradients as input and directly predicts the projection matrix $P$, avoiding the SVD computation entirely. This would reduce both the computational overhead of periodic SVD and the memory overhead of storing $P$ (if the predictor is smaller than $m \times r$). The challenge is that the predictor must learn to track the evolving gradient subspace — a meta-learning problem where the training signal comes from the downstream optimization performance. This is speculative but directly motivated by the paper's identification of SVD overhead and projection matrix memory as limitations (Section 4.2: "we can further reduce the memory cost of projection matrices by quantization and efficient parameterization, which we leave for future work").

Negative result: stress-testing GaLore on architectures that violate the reversible network conditions. The paper's theory defines a clear boundary for when GaLore should work. A valuable negative result would be to deliberately test architectures that violate the theoretical conditions: add bias terms to all linear layers (breaking the reversible network assumption that forward and backward passes have the same linear structure), use non-reversible activations (e.g., sigmoid, tanh, GELU with the erf approximation that cannot be expressed as $L(\mathbf{x}; W)\mathbf{x}$), or use architectures with dynamic routing (mixture-of-experts) where the gradient form changes discontinuously. Train GaLore on these architectures at moderate scale (e.g., 130M parameters) and measure whether gradient stable rank remains high throughout training and whether GaLore degrades relative to full-rank training. If GaLore still works, it suggests the theoretical conditions are sufficient but not necessary, and gradient low-rankness is more universal than the proofs indicate. If GaLore fails, it provides a concrete negative result that maps the boundary of applicability and guides practitioners on which architectures to avoid.

Practical Applications and Downstream Use Cases

Consumer-GPU pre-training for academic labs and individual researchers. The paper's headline result — LLaMA 7B pre-training in 22 GB, fitting on an NVIDIA RTX 4090 (24 GB) — directly enables a new class of research. An academic lab with a handful of consumer GPUs (or even a single high-end workstation) can now pre-train and study 7B-parameter models from scratch, rather than being limited to fine-tuning existing checkpoints. The practical setup: 8-bit GaLore with per-layer weight updates, rank $r = 1024$, subspace change frequency $T = 200$, token batch size 256 (or up to 4096 with activation checkpointing), scaled across multiple GPUs with data parallelism (low inter-GPU bandwidth requirement). The paper estimates optimizer state memory at 5.0 GB vs. 14.6 GB for 8-bit Adam (65.5% reduction, Section 5.5), and total training memory at 22 GB vs. 60 GB for BF16 Adam baseline (63.3% reduction). The throughput overhead is ~15.5% compared to 8-bit Adam at 1B scale (Table 11: 1019 vs. 1205 tokens/second with per-layer updates), which is modest compared to the memory savings. The key practical consideration is training time: at ~1000 tokens/second on a single consumer GPU, pre-training a 7B model on 100B tokens would take ~1157 GPU-days — practical only with data parallelism across multiple GPUs. The paper explicitly envisions this as "elastic training" across low-bandwidth consumer hardware (Section 5.5), where the small per-GPU batch size and data-parallel communication pattern make training feasible across geographically distributed or consumer-grade clusters.

Memory-constrained fine-tuning of large models without adaptor overhead. GaLore's fine-tuning results (Table 4) show that it matches or slightly outperforms LoRA on GLUE (85.89 vs. 85.61 average at rank 4) while using slightly less memory (253M vs. 257M parameter equivalents). The structural advantage is that GaLore does not need to store separate adaptor weights $B$ and $A$ alongside frozen weights $W_0$ — the full weight $W$ is updated directly, and the only additional memory is for the projection matrices $P$ and the compact optimizer states. This matters for production deployments where many fine-tuned variants of a base model must be served simultaneously: with LoRA, each variant requires storing a separate $B$ and $A$ (and potentially their optimizer states if further fine-tuning is needed); with GaLore, the model checkpoint is simply the updated weights $W$, identical in structure to the base model. The paper's fine-tuning experiments span RoBERTa-Base (GLUE), BERT-Base (SQuAD), and Gemma-2B/Phi-2/LLaMA-7B (OASST, Belle), demonstrating the method works across model families and tasks. The practical workflow: load a pre-trained model, fine-tune with GaLore at rank 4–128 (depending on available memory and task complexity), save the updated weights as a standard checkpoint with no adaptor artifacts.

Reducing carbon footprint and energy costs of LLM pre-training. The paper explicitly frames GaLore as an environmental intervention (Impact Statement): "By enabling the training of larger models on hardware with lower memory, our approach helps to minimize energy consumption and carbon footprint associated with training LLMs." The mechanism is twofold. First, GaLore enables training on fewer GPUs (or lower-power GPUs) by reducing per-GPU memory requirements, which reduces total system power draw. Second, GaLore's compatibility with data parallelism (as opposed to model parallelism) means that when multiple GPUs are used, the inter-GPU communication cost is lower — data parallelism only requires gradient averaging, while model parallelism requires communicating activations and gradients for individual layers. For a 7B model trained across 8 consumer GPUs with GaLore (each handling a micro-batch of the data-parallel split), the total energy consumption could be substantially lower than training the same model across 4 datacenter GPUs with model parallelism and full-rank optimizers. The paper does not provide energy measurements, but the memory numbers (63.3% reduction from BF16 baseline to 8-bit GaLore, Section 5.5) provide the foundation for such an analysis. A practical deployment would instrument a GaLore training run with power monitoring and compare total Joules consumed vs. a full-rank training run achieving the same perplexity.

Elastic distributed training on heterogeneous, low-bandwidth hardware. The paper explicitly envisions "elastic data distributed training on low-bandwidth consumer-grade hardware" (Section 7) as an application enabled by GaLore's small per-GPU memory footprint and data-parallel communication pattern. The scenario: a group of researchers pools their consumer GPUs (mix of RTX 3090s, 4090s, A5000s, etc.) over a network connection, and jointly pre-trains a 7B model using data parallelism with GaLore. Each GPU handles a fraction of the batch, computes gradients using GaLore in its local memory budget, and communicates only gradient averages (not model parameters or activations) — a communication volume of ~14 GB per step (the size of the full model's gradients), which is feasible even on consumer-grade network connections if gradient accumulation steps are used. The paper does not implement this, but the memory measurements (22 GB on a single RTX 4090, Figure 4) and the throughput numbers (~1000 tokens/second at 1B scale, Table 11) provide the feasibility argument. A practical implementation would need to handle heterogeneous GPU speeds (gradient accumulation with dynamic batch sizing per GPU) and fault tolerance (if a consumer GPU drops offline), but GaLore provides the memory precondition that makes the scenario viable — without it, a single consumer GPU cannot hold a 7B model's training state at all.

When to Prefer This Method

The paper explicitly positions GaLore against full-rank training and LoRA/ReLoRA, with clear tradeoff conditions derived from the empirical results (Tables 2, 3, 4; Figures 3, 4):

• Prefer GaLore over full-rank training when the available GPU memory is insufficient for full Adam optimizer states (which require ~28 GB for a 7B model in BF16, plus 14 GB for weights and 14 GB for gradients). At any model scale where the 65.5% optimizer state reduction (from GaLore projection, measured in Section 5.5) brings memory within hardware limits, GaLore enables training that would otherwise be impossible without model parallelism, offloading, or checkpointing. The throughput cost is ~15.5% at 1B scale (Table 11), which is modest compared to offloading or checkpointing overheads. The paper shows this threshold concretely: LLaMA 7B training fits in 22 GB with GaLore vs. 60 GB without (Figure 4) — a difference between consumer-grade (RTX 4090) and datacenter-grade (A100 80GB) hardware.

• Prefer GaLore over LoRA/ReLoRA when pre-training from scratch. Table 2 shows that LoRA and ReLoRA degrade severely at larger scales (LoRA: 19.21 perplexity at 1B vs. GaLore's 15.64; ReLoRA: 18.33), because static low-rank weight constraints cannot capture what the model needs to learn from random initialization. The paper explicitly evaluates ReLoRA without full-rank warmup "for a fair comparison," and the results show that even without warmup, GaLore substantially outperforms ReLoRA. If pre-training is the goal and memory is limited, GaLore is the clear choice over any static low-rank weight method.

• Prefer GaLore over LoRA when fine-tuning and adapter-free deployment is desired. Both methods achieve comparable GLUE scores (Table 4: 85.89 for GaLore rank-4 vs. 85.61 for LoRA rank-4), but GaLore produces a standard model checkpoint (updated weights only) with no adaptor storage overhead, slightly lower memory (~4 MB less for RoBERTa-Base at rank 4), and comparable or better performance on most tasks. If the fine-tuned model will be deployed in a setting where loading separate LoRA adaptors is cumbersome or where multiple fine-tuned variants must coexist without adaptor management infrastructure, GaLore is preferable.

• Prefer LoRA when the base model weights must remain frozen and shared across many tasks. LoRA's architecture (frozen $W_0$ + task-specific $BA$) naturally enables multi-task serving where a single base model is loaded and task-specific adaptors are swapped. GaLore updates the weights directly, so each fine-tuned variant requires a separate full weight copy. For serving scenarios like "thousands of personalized LoRA adaptors on a single base model" (Sheng et al., 2023), LoRA remains the better choice because GaLore does not provide adapter-based sharing.

• Prefer full-rank training (without GaLore) when GPU memory is abundant and convergence speed is the priority. The paper does not claim GaLore outperforms full-rank training in quality — it aims to match it while using less memory. Table 2 shows a small perplexity gap at 1B (15.64 vs. 15.56 for full-rank) and Table 3 shows a 0.04 gap at 7B vs. 8-bit Adam. If memory is not a constraint (e.g., training on a cluster of A100 80GB GPUs where the full 60 GB fits comfortably), the additional hyperparameters ($r$, $T$, $\alpha$) and the ~15.5% throughput overhead (Table 11) are not justified by the minimal quality gap. Full-rank training with standard Adam remains the simpler, more thoroughly validated choice when hardware permits.

• Prefer 8-bit GaLore (rather than BF16 GaLore) when targeting consumer GPUs or maximizing memory savings. The paper's 7B consumer GPU result (22 GB, Section 5.5) requires the combination of GaLore projection + 8-bit quantization + per-layer weight updates. BF16 GaLore alone reduces optimizer state memory but not enough to fit 7B in 24 GB (Figure 4 shows BF16 GaLore at ~42 GB vs. 8-bit GaLore at ~22 GB). For the specific use case of consumer-GPU pre-training, 8-bit GaLore is not an option — it is the only configuration that fits.