Read More

ArXiv: 2307.05695

🎯 Pitch

ReLoRA shows that pre-training large transformers does not actually require high-rank weight updates throughout — it’s sufficient to periodically merge low-rank adapters and reset a pruned subset of optimizer states, recovering ≥95% of full-rank perplexity on 1.3B models while trimming trainable parameters by ~5× and cutting GPU memory up to 5.5GB. This works because each restart samples a fresh low-rank subspace that, when accumulated, reconstructs a high-rank solution, redefining what we thought was necessary for large-scale training.

---

1. Executive Summary

This paper introduces ReLoRA, a parameter-efficient pre-training method that trains high-rank networks through a sequence of low-rank updates — specifically, by periodically merging LoRA adapter weights into the frozen backbone, resetting optimizer states via magnitude pruning, and re-warming the learning rate via a jagged cosine schedule. Evaluated on transformer language models up to 1.3B parameters trained on the C4 dataset, ReLoRA achieves perplexity competitive with full-rank training (17.27 vs. 16.83 at 1.3B scale) while using only a fraction of trainable parameters (250M vs. 1.3B), yielding a 9–40% training speedup and up to 5.5Gb RAM savings per GPU depending on hardware. The singular value spectrum analysis establishes that ReLoRA genuinely learns a high-rank weight update — closer to full-rank training than to standard LoRA — establishing that locally low-rank training suffices for pre-training only when combined with periodic restarts that span different low-rank subspaces across the trajectory.

2. Context and Motivation

The Core Problem: Pre-Training Costs Are Prohibitive, and Overparameterization Remains Poorly Understood

The fundamental tension this paper addresses has two intertwined dimensions — one practical, one theoretical — and both stem from the same empirical reality: modern neural network training is dominated by the "stack more layers" paradigm (Krizhevsky et al., 2012; He et al., 2016; Kaplan et al., 2020). The practical dimension is straightforward: training large models is becoming economically infeasible for most research groups. The paper opens by tracing the evolution of what constitutes a "large" network — from ~100M parameters (Simonyan and Zisserman, 2015; Radford et al., 2018) to hundreds of billions (Brown et al., 2020; Chowdhery et al., 2022) — and notes that the computational costs have grown exponentially along with model size. Training a model like LLaMA-7B (Touvron et al., 2023) "still requires hundreds of GPUs," and training RETRO (Borgeaud et al., 2022) demands infrastructure capable of searching over trillions of tokens. This puts multi-billion-parameter pre-training out of reach for the vast majority of academic labs and smaller industry teams.

The theoretical dimension is more subtle but equally important: we do not understand why overparameterization is necessary in the first place. The paper explicitly states this in its opening paragraph:

"the necessity to train models which can have orders of magnitude more parameters than the training examples... is poorly understood theoretically."

This isn't merely an academic curiosity. If overparameterization were well-understood — if we knew precisely which parameters matter and why — then we might be able to achieve comparable performance without the enormous redundancy that current architectures exhibit. The Lottery Ticket Hypothesis (Frankle et al., 2019) suggests such redundancy exists: within a large, randomly-initialized network, there exist sparse sub-networks ("winning tickets") that, when trained in isolation, can match the full network's performance. This implies that most parameters in a large model may be unnecessary — but identifying those winning tickets before training remains unsolved, and the existence of lottery tickets does not tell us how to train the sparse sub-network from scratch without first training the dense model (or at least finding the ticket through iterative magnitude pruning, which itself requires partial training).

The paper positions itself at the intersection of these two problems: if we could train only a subset of parameters at any given time while still achieving the representational capacity of a full-rank weight update, we could simultaneously reduce training costs and shed light on why overparameterization helps. This is the motivation for exploring parameter-efficient pre-training: not just fine-tuning a frozen pre-trained model (which LoRA and other PEFT methods already do effectively), but actually pre-training from scratch using parameter-efficient methods.

The Gap: Parameter-Efficient Fine-Tuning Works, but Parameter-Efficient Pre-Training Remains Open

The paper observes a striking asymmetry in the literature. On one side, parameter-efficient fine-tuning (PEFT) has been a major success story. Methods like LoRA (Hu et al., 2022), adapters, prompt tuning, and their variants have enabled fine-tuning of billion-parameter models on consumer hardware. The key insight from Aghajanyan et al. (2021) — that pre-training reduces the intrinsic dimensionality of the update needed to learn a downstream task — provides a theoretical basis: the larger the pre-trained model, the lower-rank the fine-tuning update can be while maintaining performance. This means that during fine-tuning, the model only needs to explore a small subspace of its full parameter space.

On the other side, this same logic does not obviously extend to pre-training. During pre-training from scratch, the network starts from a random initialization and must learn everything — token representations, grammatical structure, factual knowledge, reasoning patterns — from the raw data distribution. There is no pre-existing structure to exploit; the model must construct its representational capacity from nothing. Intuitively, one might expect that this requires full-rank updates, since the model needs to explore the full parameter space to find good solutions. The paper confirms this intuition empirically: standard LoRA applied to pre-training from scratch fails dramatically. In Table 2, LoRA alone achieves perplexity of 47.44 on the 60M model versus 33.81 for full-rank training — a massive gap that only widens with model scale (e.g., 57.11 vs. 18.66 at 350M). This is the central empirical puzzle: why does LoRA — which works beautifully for fine-tuning — completely break down for pre-training?

The paper's answer to this puzzle is its core conceptual contribution. LoRA, by construction, constrains the weight update to a single fixed low-rank subspace throughout the entire training trajectory. The product $W_A W_B$ has rank at most $r$ (where $r$ is the LoRA rank, typically 8–128), and because $W_A$ and $W_B$ are never merged and re-initialized, all updates across all training steps live in this same $r$-dimensional subspace. For fine-tuning, where the model starts close to a good solution and only needs small adjustments, this constraint is beneficial — it acts as a regularizer that prevents catastrophic forgetting. For pre-training from scratch, however, a single low-rank subspace is insufficient to capture the full complexity of the learning trajectory. The model needs to explore different directions at different stages of training, and a fixed low-rank bottleneck prevents this exploration.

Where Prior Low-Rank Training Methods Fall Short

The paper acknowledges a body of prior work on low-rank neural network training (Idelbayev and Carreira-Perpinan, 2020; Jaderberg et al., 2014; Schotthöfer et al., 2022; Sui et al., 2023; Kamalakara et al., 2022; Lin et al., 2020; Yuan et al., 2021; Zhao et al., 2023), but identifies specific limitations that prevent these methods from addressing the pre-training gap:

1. Most prior methods are CNN-specific or architecture-dependent. Idelbayev and Carreira-Perpinan (2020) and Jaderberg et al. (2014) focus on convolutional architectures and exploit structural properties (e.g., spatial redundancy in filters) that do not transfer to transformers. The transformer's self-attention mechanism has fundamentally different rank properties — Bhojanapalli et al. (2020) demonstrated that insufficient rank in key and query projections actually bottlenecks transformer performance, meaning that any low-rank training method for transformers must be careful not to overly constrain the attention mechanism.

2. Existing methods have not been evaluated at scale on large transformers. The paper notes that prior low-rank training methods "do not scale well, or have not been evaluated on large transformers with hundreds of millions of parameters, which can benefit greatly from efficient training." This is a critical point: a method that works on a 10M-parameter CNN or a small transformer may exhibit entirely different dynamics at the 350M–1.3B scale where the efficiency gains matter most. The paper's experiments span five model sizes from 60M to 1.3B parameters, systematically testing whether the approach degrades or improves with scale.

3. Fixed low-rank factorization fundamentally limits representational capacity. Methods that decompose weight matrices once and train the factors (like standard LoRA applied naively to pre-training) suffer from the single-subspace limitation described above. Kamalakara et al. (2022) explored low-rank training of deep networks but did not incorporate restarts or subspace changes during training, which the ReLoRA paper identifies as the critical missing ingredient.

4. The "locally low-rank" property of training trajectories was not exploited. While several prior works observed that neural network training exhibits low-rank properties — Aghajanyan et al. (2021) showed that fine-tuning updates are low-rank, Arora et al. (2019) found that SGD is biased toward low-rank solutions, and the Lottery Ticket Hypothesis (Frankle et al., 2019) implies that sparse (and thus effectively low-rank) sub-networks can be trained successfully — none of these observations had been operationalized into a pre-training method. The ReLoRA paper's key synthesis is the insight that training is locally low-rank: for any sufficiently short interval of the training trajectory, the update can be well-approximated by a low-rank matrix, but the direction of this low-rank subspace changes over the course of training. This is what the paper terms "locally low-rank training" (Section 2), and it directly motivates the restart mechanism: each ReLoRA segment learns a different low-rank update, and their sum yields a high-rank total update.

How This Paper Positions Itself

The paper positions ReLoRA as a bridge between two previously disconnected lines of work: parameter-efficient fine-tuning (which succeeds because fine-tuning is inherently low-rank) and full pre-training (which appears to require high-rank updates but may be decomposable into a sequence of low-rank phases). The central hypothesis is stated in Section 2:

"Multiple studies suggest that neural network training is either completely low-rank or has multiple phases with initially high-rank and subsequent low-rank training."

If training is locally low-rank — meaning that within any short window of a few thousand steps, the gradient updates predominantly occupy a low-dimensional subspace — then it should be possible to train using low-rank adapters that are periodically merged and reset, with each reset allowing the model to explore a different low-rank subspace. The sum of these low-rank updates across resets yields a high-rank total update (Equation 2 in the paper), matching the representational capacity of full-rank training while only ever training a small number of parameters at any given time.

This positioning is significant because it reframes the problem. Rather than asking "can we train a low-rank network that matches a full-rank network?" (which prior work suggests is difficult), the paper asks "can we train a full-rank network using a sequence of low-rank updates?" The key insight is that the total update matters — not any individual step. This connects conceptually to the Lottery Ticket Hypothesis with rewinding (Frankle et al., 2019), where the authors found that early training is critical for establishing the right "initialization" from which a sparse sub-network can be successfully trained. ReLoRA's warm-start phase serves a similar purpose: a short period of full-rank training at the beginning establishes a good initialization, after which low-rank updates can effectively continue the optimization. The paper explicitly acknowledges this parallel:

"Similar to the lottery ticket hypothesis with rewinding, ReLoRA employs a full-rank training warm start before transitioning to ReLoRA."

The practical framing is also important. The paper does not claim that ReLoRA is theoretically optimal or that it solves the overparameterization puzzle. Instead, it presents ReLoRA as an empirical demonstration that parameter-efficient techniques can be adapted for pre-training, opening a new research direction. The efficiency gains — 5.5Gb RAM savings per GPU, 9–40% training speedup depending on hardware — are substantial enough to matter for resource-constrained settings, but the paper is transparent that these are hardware-dependent and that the largest relative gains appear on lower-bandwidth hardware where the reduced optimizer state overhead is most impactful (Table 7).

The Specific Mechanisms That Enable ReLoRA

Though covered in detail in the Technical Approach section, it's worth noting in the motivation section why ReLoRA's components are non-trivial additions over standard LoRA. The paper's ablation studies (Table 6) reveal that simply adding restarts to LoRA — merging $W_A$ and $W_B$ and re-initializing them periodically — does not work on its own. Without an optimizer reset, the Adam momentum terms ($\beta_1 \approx 0.9$, $\beta_2 \approx 0.999$) continue to guide the newly initialized parameters in the same direction as the previous parameters, defeating the purpose of the restart. Without a jagged learning rate schedule (Figure 2), the model diverges after the reset because the optimizer state disruption causes loss spikes. The combination of (1) partial optimizer state pruning via magnitude, (2) jagged cosine learning rate schedule with re-warmup, and (3) full-rank warm start is what makes the restart mechanism viable — and the paper demonstrates through ablation that each component is necessary for the final performance.

This is an important motivational point: the reason parameter-efficient pre-training remained unsolved despite the success of LoRA for fine-tuning is not simply that no one thought to try restarts. The restart mechanism interacts with optimization dynamics in subtle ways that require careful mitigation, and identifying those mitigations is a genuine technical contribution.

3. Technical Approach

This is primarily an empirical methods paper whose core idea is that full-rank neural network training can be decomposed into a sequence of low-rank updates by periodically merging and resetting low-rank adapters, provided the optimization procedure is carefully modified to prevent the optimizer from re-using stale momentum directions across resets.

3.1 Reader Orientation

ReLoRA is a training algorithm that modifies how a neural network's weight matrices are updated during pre-training — instead of updating all parameters at every step (full-rank training), it only trains a small set of low-rank adapter matrices at any given time, but periodically merges these adapters into the frozen backbone weights and resets them, allowing the next adapter to learn in a different low-rank subspace. The system solves the problem that standard low-rank training (LoRA applied to pre-training from scratch) fails catastrophically because constraining updates to a single fixed low-rank subspace across the entire training trajectory prevents the model from exploring the diverse directions needed to learn from raw data; ReLoRA's solution shape is to make the training locally low-rank while keeping the total update high-rank through repeated merge-and-reset cycles, each exploring a different subspace.

3.2 Big-Picture Architecture (Diagram in Words)

The ReLoRA training pipeline has five major components that operate in sequence:

1. Full-Rank Warm-Start Phase — The model is trained conventionally (all parameters updated) for a fixed number of steps to establish a good initialization, analogous to finding a lottery ticket initialization before pruning.

2. LoRA Wrapping — After the warm start, every linear layer in the transformer (attention projections, feed-forward layers) is replaced with a frozen weight matrix plus a trainable low-rank adapter ($W_A W_B$), while embeddings and normalization layers continue to be trained full-rank.

3. Low-Rank Training Segments — The model trains using only the adapter parameters (plus embeddings/norms) for a fixed interval (typically 2,000–5,000 steps), during which all updates are constrained to the current adapter's low-rank subspace.

4. Merge-and-Reinit (Restart) — At the end of each segment, the adapter weights are merged into the frozen backbone ($W \leftarrow W + s W_A W_B$), the adapter matrices are re-initialized (Kaiming init for $W_A$, zeros for $W_B$), 90–99% of the Adam optimizer states are pruned by magnitude, and the learning rate is dropped to zero with a subsequent warm-up back to the cosine schedule.

5. Jagged Cosine Learning Rate Scheduler — The learning rate follows a standard cosine decay but resets to zero at each restart and warms up over 50–100 steps, creating a sawtooth pattern superimposed on the cosine envelope.

Information flows as follows: The raw text data enters the model → the model processes it using the current low-rank adapter configuration → gradients flow only through the adapter parameters (frozen backbone weights are not updated) → after a fixed number of steps, the adapters are merged, re-initialized, and the optimizer state is pruned → the next segment begins with a new learning rate warm-up → after the final segment, all adapters have been merged, leaving a single high-rank weight matrix equivalent to the full-rank trained model.

3.3 Roadmap for the Deep Dive

• First, the mathematical foundation — the rank-of-sum property and how it motivates ReLoRA's core mechanism — because understanding why repeated low-rank updates can yield a high-rank total update is essential to understanding the entire approach.
• Second, the standard LoRA formulation that ReLoRA builds upon, since ReLoRA is a modification of LoRA and readers need to understand the base mechanism before the modifications.
• Third, the restart mechanism in detail — what physically happens during a merge-and-reinit, why naive restarts fail, and how optimizer state pruning and jagged scheduling solve the failure modes — because this is ReLoRA's primary technical innovation.
• Fourth, the warm-start phase and its justification, connecting to the Lottery Ticket Hypothesis with rewinding.
• Fifth, the computational efficiency argument — why training fewer parameters at a time saves memory and bandwidth even though the total parameter count hasn't changed — since this is the practical motivation.
• Sixth, the locally low-rank training hypothesis that provides theoretical motivation for why ReLoRA should work.

3.4 Detailed, Sentence-Based Technical Breakdown

The Mathematical Foundation: Rank of Sum

ReLoRA is built on a simple linear algebra fact: the rank of a sum of matrices can be greater than the rank of any individual summand. Formally, for any two matrices $A$ and $B$ of the same dimensions:

$\text{rank}(A + B) \leq \text{rank}(A) + \text{rank}(B)$

where $\text{rank}(\cdot)$ is the number of linearly independent rows or columns in the matrix, $A$ and $B$ are the two matrices being summed, and the inequality is tight (there exist $A$ and $B$ such that $\text{rank}(A + B) = \text{rank}(A) + \text{rank}(B)$).

What it computes: This inequality describes the maximum possible rank achievable by adding two matrices. If $A$ has rank $r_A$ and $B$ has rank $r_B$, their sum can have rank anywhere from $|r_A - r_B|$ (if the column spaces overlap maximally) to $r_A + r_B$ (if the column spaces are disjoint — the matrices contribute information in entirely different directions).

Why this form matters for ReLoRA: The key implication is that repeated addition of low-rank updates can build a high-rank total update, provided each new low-rank update contributes information in a subspace different from all previous updates. If LoRA trains a single low-rank adapter throughout the entire training trajectory, the total update has rank at most $r$ (the adapter rank). But if the adapter is periodically merged and reset — and if each new adapter learns in a different subspace — then the sum of $N$ adapter updates can have rank up to $N \times r$. This is the mathematical basis for ReLoRA's claim that it can train a high-rank network through low-rank updates. The paper explicitly formalizes this in Equation 2:

$\Delta W = \sum_{t=0}^{T_1} \delta W_t + \sum_{t=T_1}^{T_2} \delta W_t + \cdots + \sum_{t=T_{N-1}}^{T_N} \delta W_t = s W^1_A W^1_B + s W^2_A W^2_B + \cdots + s W^N_A W^N_B$

where $\Delta W$ is the total weight update from the start of LoRA training to the end, $T_i$ are the step indices at which restarts occur, $N$ is the total number of restart segments, $s \in \mathbb{R}$ is the fixed LoRA scaling factor (typically $1/r$), $W^i_A \in \mathbb{R}^{d_{\text{in}} \times r}$ is the $i$-th segment's A-matrix (the low-rank projection from the input dimension to the rank-$r$ bottleneck), and $W^i_B \in \mathbb{R}^{r \times d_{\text{out}}}$ is the $i$-th segment's B-matrix (the projection from the bottleneck to the output dimension).

What it computes: The total trained weight update $\Delta W$ is the sum of $N$ independent low-rank products $W^i_A W^i_B$, each of which individually has rank at most $r$, but whose sum can have rank up to $N \times r$ (or capped at $\min(d_{\text{in}}, d_{\text{out}})$, the maximum possible rank of the full weight matrix). This is the operational definition of "training a high-rank network through low-rank updates": each segment contributes a rank-$r$ update, and because different segments can learn in different subspaces, the accumulated update can span the full rank of the weight matrix.

Why this form: The decomposition into a sum of rank-$r$ products directly mirrors the standard LoRA formulation $\delta W = s W_A W_B$, but with the crucial difference that the sum runs over multiple independently initialized adapter pairs. Without this sum — i.e., standard LoRA with no restarts — $\Delta W = s W_A W_B$ is constrained to rank at most $r$, which the paper's experiments show is insufficient for pre-training (LoRA alone achieves perplexity 34.17 vs. 23.65 for full-rank on 130M models, per Table 6). The sum-of-restarts formulation removes this constraint while maintaining the memory efficiency of only training $r \times (d_{\text{in}} + d_{\text{out}})$ parameters at any given time.

A critical practical subtlety: the merged weights $W + \Delta W$ at the end of training are mathematically equivalent to a standard fully-trained weight matrix — there is no remaining low-rank factorization, no adapter overhead at inference time. The adapters are training-time artifacts that exist only to constrain the optimization trajectory; after the final merge, the model is a standard dense network.

Standard LoRA Formulation (The Base That ReLoRA Modifies)

ReLoRA builds directly on LoRA (Low-Rank Adaptation, Hu et al., 2022), so understanding LoRA's mechanics is prerequisite. LoRA is a parameter-efficient fine-tuning method that modifies any linear layer parametrized by a weight matrix $W \in \mathbb{R}^{m \times n}$ by freezing $W$ and adding a trainable low-rank update:

$\delta W = s W_A W_B$

where $s \in \mathbb{R}$ is a fixed scaling factor (typically $1/r$ where $r$ is the rank), $W_A \in \mathbb{R}^{m \times r}$ is the "A-matrix" that projects the $m$-dimensional input down to the $r$-dimensional bottleneck, $W_B \in \mathbb{R}^{r \times n}$ is the "B-matrix" that projects from the bottleneck up to the $n$-dimensional output, and $\delta W \in \mathbb{R}^{m \times n}$ is the effective weight update added to $W$ during the forward pass.

What it computes: During the forward pass, the layer computes $y = xW + x(s W_A W_B) = x(W + s W_A W_B)$, where $x \in \mathbb{R}^{m}$ is the input vector and $y \in \mathbb{R}^{n}$ is the output. The first term $xW$ is the frozen pre-trained computation; the second term $x(s W_A W_B)$ is the trainable low-rank adjustment. During the backward pass, gradients flow only through $W_A$ and $W_B$; $W$ receives no updates.

Why this form: The decomposition $W_A W_B$ reduces the number of trainable parameters from $m \times n$ (full fine-tuning) to $r \times (m + n)$ (LoRA). For a typical transformer where $m = n = 4096$ (hidden size) and $r = 128$, this reduces trainable parameters from ~16.8M to ~1.05M per weight matrix — a ~16× reduction. The scaling factor $s = 1/r$ is chosen so that the initial contribution of the adapter is near zero (since $W_A$ is initialized with small random values via Kaiming initialization and $W_B$ is initialized to zeros, the product $s W_A W_B$ starts at zero regardless of $r$), meaning the model begins fine-tuning from the pre-trained weights without perturbation.

The initialization convention — Kaiming (He) initialization for $W_A$, zeros for $W_B$ — is critical and inherited by ReLoRA. When $W_B = 0$, the adapter contributes nothing on the first forward pass, so the model's output is identical to the frozen backbone. This ensures a smooth start: the adapter learns from zero rather than from a random perturbation that could destabilize early training. During ReLoRA restarts, the same initialization is used, ensuring that each new segment also starts from the current merged weights without perturbation.

The Restart Mechanism: Why Naive Restarts Fail

The core innovation of ReLoRA is the periodic merge-and-reinit, but the paper discovered that simply merging the adapter and re-initializing it — what one might naively call "restarting LoRA" — does not work. The ablation in Table 6 tells the story numerically (130M model):

• LoRA alone (no restarts, no warm start): 34.17 perplexity
• Restarts added (merge and reinit, but no optimizer reset or jagged schedule): 34.25 perplexity — essentially identical to LoRA, no improvement
• Restarts + optimizer reset (but no jagged schedule): model diverges
• Restarts + jagged schedule (but no optimizer reset): 34.29 perplexity — also no improvement

The reason the second configuration fails to improve is subtle and involves the Adam optimizer's internal state. Adam (Kingma and Ba, 2015) maintains two running averages for each parameter: the first moment $m_t$ (exponential moving average of gradients, controlled by $\beta_1$) and the second moment $v_t$ (exponential moving average of squared gradients, controlled by $\beta_2$). The update rule is:

$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$ $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$ $\theta_t = \theta_{t-1} - \eta \frac{m_t}{\sqrt{v_t} + \epsilon}$

where $g_t$ is the gradient at step $t$, $\beta_1 \in [0, 1)$ is the decay rate for the first moment (typically 0.9), $\beta_2 \in [0, 1)$ is the decay rate for the second moment (typically 0.999), $\eta$ is the learning rate, $\epsilon$ is a small constant for numerical stability, and $\theta_t$ are the parameters at step $t$.

What the Adam update computes: Each parameter update is proportional to $m_t / \sqrt{v_t}$, which is a signal-to-noise-ratio-scaled estimate of the recent gradient direction. The $m_t$ term provides momentum — the optimizer continues pushing parameters in directions that have been consistently beneficial over recent steps. The $v_t$ term provides adaptive per-parameter learning rates — parameters with consistently large gradients get smaller effective step sizes.

Why Adam state causes naive restarts to fail: When ReLoRA merges $W_A$ and $W_B$ and re-initializes them, the parameters are reset, but Adam's $m_t$ and $v_t$ for those parameters are not automatically reset — they retain values accumulated from the previous segment. The paper explains this directly:

"Adam's $\beta_1$ and $\beta_2$ are usually very high 0.9–0.999. This means that after the merge-and-reinit, continuing to use old gradient moments for $W^2_A$ will guide it in the same direction as $W^1_A$ and optimize the same subspace."

The consequence is that the newly initialized $W^2_A$ and $W^2_B$, despite having fresh random/zero weights, are steered by optimizer momentum to converge to approximately the same solution as $W^1_A$ and $W^1_B$. The total update $sW^1_A W^1_B + sW^2_A W^2_B$ ends up having rank only slightly above $r$, because the two adapters learned overlapping subspaces. This defeats the entire purpose of restarts, which is to accumulate rank across segments.

The Solution: Partial Optimizer State Reset via Magnitude Pruning

To break the optimizer's momentum across restarts, ReLoRA performs a partial reset of the optimizer state at each merge-and-reinit. The paper specifies this as:

"ReLoRA performs a partial reset of the optimizer state during merge-and-reinit via magnitude pruning."

The procedure (Algorithm 1, lines 16–17) is:

$M_{W_A} \leftarrow \text{prune}(M_{W_A}) \quad \text{and} \quad V_{W_A} \leftarrow \text{prune}(V_{W_A})$

where $M_{W_A}$ and $V_{W_A}$ are the Adam first and second moment buffers for the A-matrix parameters, and $\text{prune}(\cdot)$ zeros out a specified fraction of entries based on their magnitude (smallest values are pruned first). The paper reports that pruning rates above 90% work well, and the ablation in Appendix B (Table 9) shows that even 99% pruning does not degrade performance (QNLI accuracy with 85% pruning: 94.51; with 99% pruning: 94.56). The B-matrix optimizer states are also pruned, and both A and B matrices are re-initialized (Kaiming init for A, zeros for B).

What this does operationally: Before the first step of the new segment, a large fraction (90–99%) of the stored gradient history is erased. The optimizer effectively "forgets" most of what it learned about good update directions in the previous segment. The remaining 1–10% of un-pruned optimizer state provides a small amount of continuity — enough to prevent the loss from spiking catastrophically — but not enough to force the new adapter into the same subspace as the old one. The new adapter is free to explore a different low-rank subspace because the optimizer's momentum has been largely reset.

Why magnitude pruning specifically: The paper does not provide extensive justification for magnitude pruning over alternative approaches (e.g., complete reset, random reset), but the intuition is straightforward: the largest-magnitude optimizer states correspond to parameters that had the strongest gradient signal in the previous segment, and preserving a small fraction of these provides just enough guidance to stabilize the transition. Complete reset causes divergence (Table 6, row 3: "Restarts + Optimizer Reset without Jagged Schedule = diverged"), because the optimizer has no memory at all and the freshly initialized parameters receive large, noisy gradients that cause loss spikes. Magnitude pruning is a middle ground: enough reset to allow subspace change, enough continuity to maintain stability.

The paper's fine-tuning ablations (Table 9) confirmed that pruning percentage is not a sensitive hyperparameter — "the model's performance remains largely unaffected even when 99% of the optimizer states are reset." This robustness is practically important because it means practitioners do not need to carefully tune the pruning threshold; any value above 90% works.

The Jagged Cosine Learning Rate Schedule

Even with optimizer state pruning, the paper found that the loss can diverge after a restart if the learning rate is not carefully managed. The solution is a jagged cosine learning rate schedule, illustrated in Figure 2, which modifies a standard cosine decay schedule by inserting learning rate resets at each restart:

"To avoid loss diverging after an optimizer reset it also sets the learning rate to 0 with a subsequent warm-up."

The schedule is defined as follows (Section 2, final paragraph before "Enhancing computational efficiency"):

1. The base schedule follows a standard cosine decay from the initial learning rate $\eta_{\text{max}}$ to a minimum value (following Touvron et al., 2023).
2. At each restart step (every $q$ steps, typically $q = 2000$ to $5000$), the learning rate is instantly set to zero.
3. Over the next 50–100 steps, the learning rate linearly warms up from zero back to the value it would have had under the unmodified cosine schedule at that point in training.
4. After the warm-up, the learning rate continues following the cosine decay until the next restart.

Why this form: The jagged schedule serves two purposes. First, dropping the learning rate to zero immediately after the optimizer state reset prevents the newly initialized parameters from taking large steps based on the (now inaccurate, because the optimizer state was pruned) gradient estimates. The model gets 50–100 steps of very small updates to "settle in" to the new adapter configuration before the learning rate ramps back up. Second, the re-warmup compensates for the optimizer state reset: since the optimizer's second-moment estimates $v_t$ have been largely erased, the adaptive per-parameter learning rate scaling $1/\sqrt{v_t}$ is no longer accurate. A lower learning rate during the warm-up period gives the optimizer time to rebuild reasonable $v_t$ estimates from fresh gradient statistics before full-size updates resume.

The ablation in Table 6 confirms the necessity: "Restarts and optimizer resets without a jagged schedule causes the model to diverge" (row 3, marked "diverged"). The jagged schedule also enables faster warm-up compared to training from scratch: the paper notes that "a combination of partial optimizer reset and jagged scheduler allows for a quicker warm-up, as low as 50 steps, instead of hundreds of steps required when the optimizer is initialized from scratch."

Algorithm 1: The Complete ReLoRA Procedure

The paper presents the full ReLoRA algorithm as Algorithm 1, which can be walked through step-by-step:

Phase 1: Warm-Start (Lines 1–3) The model $\theta$ (the full set of parameters including all linear layers, embeddings, and normalization layers) is trained using standard full-rank training for $t_{\text{warm}}$ steps. The Adam optimizer states $M$ and $V$ are updated normally, and the learning rate $\eta$ follows the standard schedule (no jaggedness yet). This phase establishes a good initialization from which the low-rank segments can effectively continue.

Phase 2: LoRA Wrapping (Lines 4–9) After the warm start, every linear layer in the model is transformed:

1. The current weight matrix $W^i$ (where $i$ indexes the layer) is frozen — it will not receive gradient updates for the remainder of training.
2. Two new trainable matrices are added: $W^i_A$ (initialized with Kaiming uniform initialization) and $W^i_B$ (initialized to zeros).
3. The forward pass for this layer becomes $y = xW^i + s \cdot x W^i_A W^i_B$.

Embedding layers and normalization layers (RMSNorm in this architecture) are not wrapped — they continue to be trained full-rank throughout. This is a design choice that the paper adopts from the LoRA convention: embeddings and norms typically have relatively few parameters compared to linear layers, so the memory savings from low-ranking them would be negligible, and their full-rank training may be important for maintaining representational capacity.

Phase 3: Low-Rank Training Segments with Periodic Restarts (Lines 10–22) This is the main training loop. For each training step $t$ after the warm start:

1. A standard optimizer step is taken, updating only the trainable parameters $\hat{\theta}$ (the A and B matrices for all wrapped layers, plus embeddings and norms). The frozen weights $W^i$ receive no updates. The optimizer states $M$ and $V$ are updated normally.
2. Every $q$ steps (the reset frequency), a restart is triggered:
   • Merge (Line 15): For each wrapped linear layer, the adapter weights are merged into the frozen backbone: $W^i \leftarrow W^i + s W^i_A W^i_B$. After this operation, $W^i$ now contains the accumulated update from all previous segments.
   • Reinitialize (Line 16): $W^i_A$ is reinitialized with Kaiming uniform initialization; $W^i_B$ is set to zeros. This ensures the new adapter starts from zero contribution (because $W^i_B = 0$ means $s W^i_A W^i_B = 0$).
   • Optimizer State Pruning (Line 17): The Adam first and second moment buffers for the A and B matrices are pruned: a specified fraction (e.g., 90–99%) of entries with the smallest magnitude are set to zero. The paper does not prune the optimizer states for embeddings and normalization layers (these continue to train full-rank and do not need subspace resets).
   • Learning Rate Warm-up (Line 20): The learning rate is dropped to zero and warmed up over the next 50–100 steps back to the cosine schedule.
3. Training continues with the new adapter, now exploring a different low-rank subspace than the previous segment.

Phase 4: Final Model (Line 23) After the final training step, the model parameters $\theta$ are returned. Crucially, because all adapters have been merged at each restart, the returned model is a standard dense network with no adapter overhead — the $W^i$ matrices now equal the original warm-start weights plus the sum of all adapter contributions.

Hyperparameter Configuration

The paper specifies the following key hyperparameters used across experiments (Section 3, "ReLoRA and baselines setup"):

• LoRA rank $r$: 128 across all model sizes (60M through 1.3B). The paper states this was chosen because "initial experiments showed it to have the best perplexity/memory trade-off." Section 4.1 reports a rank sweep on the 1.3B model showing "very little difference between ranks 128 (ppl. 19.16) and 512 (ppl. 19.00)," suggesting the method is not highly sensitive to this parameter. Appendix A recommends $r \in \{64, 128\}$ for all networks up to 1B.
• Reset frequency $q$: The main experiments use $q = 5000$ steps, with 3 total resets (warm start at 5,000 steps, resets at 10,000 and 15,000 steps, total training 20,000 steps). Section 4.1 tests different warm-start durations (2K, 5K, 10K steps) for the 1.3B model, with 10K performing best. Appendix A reports that "2K iterations reset rate performed consistently well in both pre-training and fine-tuning experiments."
• Optimizer pruning rate: 99% in the main experiments ("99% of the optimizer state is pruned based on magnitude"). Appendix A notes that performance is robust "as long as it's larger than 90%," with higher rates possibly giving slightly better performance.
• Learning rate warm-up after restart: 100 steps in the main experiments; the paper notes that the jagged schedule enables warm-ups "as low as 50 steps" with partial optimizer reset, versus "hundreds of steps required when the optimizer is initialized from scratch."
• Learning rate scaling: ReLoRA (and LoRA) requires "1.5–2 times larger learning rate than regular training/fine-tuning to achieve similar performance" (Appendix A). This is a crucial practical detail: the effective learning rate per parameter is lower when only a subset of parameters are trainable, because the gradient signal is distributed across fewer degrees of freedom.
• LoRA scaling factor $s$: Fixed at $1/r$ following LoRA convention (Equation 1).
• Where LoRA is applied: All attention projections (query, key, value, output) and all feed-forward network layers (up-projection, gate-projection, down-projection), but not embeddings or normalization layers. Appendix A states: "In all of our experiments, we applied LoRA/ReLoRA to all of the linear layers in the model: kqv-projection layers, FFN layers and other projections, except for logits and embeddings."

The Full-Rank Warm Start

The warm-start phase — a period of standard full-rank training before switching to ReLoRA — is empirically the single most important component. In the ablation (Table 6), adding a warm start (going from the fourth row to the fifth row) drops perplexity from 29.77 to 25.46 on the 130M model, and adding all components (going from fifth row to sixth row, which adds restarts, optimizer reset, and jagged schedule) only drops it further from 25.46 to 25.04. The warm start alone accounts for the majority of the gap between pure LoRA (34.17) and full ReLoRA (25.04).

The paper connects this to the Lottery Ticket Hypothesis with rewinding (Frankle et al., 2019):

"This demonstrates the importance of early training, similar to the concept of the lottery ticket hypothesis with rewinding."

The analogy works as follows: In the Lottery Ticket Hypothesis, a randomly initialized network contains sparse sub-networks ("winning tickets") that can be trained in isolation to match the full network's performance — but only if they are rewound to their initial values from early in training, not from random initialization. The early phase of full training establishes a good "initialization" for the sparse sub-network. Similarly, in ReLoRA, the warm-start phase establishes a good "initialization" from which low-rank adapters can effectively continue optimization. Starting ReLoRA from random initialization (no warm start) fails because the adapters need to reconstruct both the initialization and the task learning simultaneously, which a single low-rank subspace cannot accommodate.

The paper quantifies this: "we measured the perplexity of the warmed-up network, which equals 27.03. It outperforms all low-rank methods except for our final ReLoRA recipe but still demonstrates a significant difference from the final network." This means the warm start alone achieves 27.03 perplexity on the 130M model, and the subsequent ReLoRA training further improves it to 25.04 — confirming that the low-rank segments are doing meaningful learning beyond what the warm start established.

The warm-start duration is a hyperparameter: "we performed warm start for about 1/4 of the total training updates" in most experiments, but at 1.3B scale the best run used a 10K-step warm start out of 30K total steps (33%), suggesting that larger models may benefit from proportionally longer warm starts. Appendix E (Figure 13) shows that even a very short warm start (2K steps) still enables ReLoRA to significantly outperform LoRA (perplexity 23.64 vs. 25.08), though the absolute performance is lower than with longer warm starts.

Computational Efficiency: Why Fewer Trainable Parameters Saves Memory and Time

The paper's efficiency argument (Section 2, "Enhancing computational efficiency") explains why training only a subset of parameters reduces memory and improves speed, even though the total number of parameters in the model hasn't changed:

1. Optimizer state memory reduction. Adam maintains two buffers ($m_t$ and $v_t$) per trainable parameter. In standard full-rank training with a model of $P$ parameters, the optimizer state requires $2P$ values (plus the parameters themselves, so $3P$ total in float32 equivalent). In ReLoRA, only the adapter parameters (plus embeddings and norms) are trainable. If the adapters have $P_{\text{train}}$ parameters, the optimizer state requires only $2P_{\text{train}}$ values. The frozen backbone weights $P_{\text{frozen}} = P - P_{\text{train}}$ need only be stored in memory, not optimized. For the 1.3B model with 250M trainable parameters (vs. 1.3B total), the optimizer state reduction is approximately $(1.3\text{B} - 250\text{M}) \times 2 = 2.1\text{B}$ fewer stored values.

2. Larger effective batch sizes. Because less GPU RAM is consumed by optimizer states, more memory is available for activations, enabling larger micro-batch sizes. The paper reports that ReLoRA allowed "two times larger microbatch size" for the 1.3B model, which improves hardware utilization (GPUs are more efficient with larger matrix multiplications) and reduces the number of gradient accumulation steps needed to achieve a target global batch size.

3. Reduced communication bandwidth in distributed training. In data-parallel distributed training, gradients for all trainable parameters must be communicated (all-reduced) across GPUs at every step. With fewer trainable parameters, the communication volume per step is proportionally reduced. This is often the bottleneck in large-scale distributed training, so reducing communication can yield speedups even if the per-GPU computation is unchanged.

4. Potential for quantization of frozen weights. Since the frozen backbone weights $W^i$ are not updated between restarts, they can be stored in a low-precision quantized format (e.g., 4-bit or 8-bit) to further reduce memory. The paper mentions this as a future possibility (citing QLoRA, Dettmers et al., 2023) but does not implement it in the reported experiments.

The empirical memory savings are reported as 5.5Gb per GPU (from 27.8Gb to 22.3Gb) for the 1.3B model when training with the same micro-batch size. The training speed improvements vary significantly by hardware, as shown in Table 7:

• 8×A100 (high-bandwidth HBM): ReLoRA throughput is 157 ex/sec vs. 137 ex/sec full-rank — a 15% immediate speedup, but after amortizing the warm-start cost (33% of training done full-rank), the total speedup is only 9%.
• 6×A6000 Ada (mid-range): ReLoRA throughput is 124 ex/sec vs. 84 ex/sec — 48% immediate, 32% total with warm start.
• 2×RTX 3090 (consumer): ReLoRA throughput is 17.8 ex/sec vs. 8.8 ex/sec — 102% immediate, 51% total with warm start.

The paper attributes this hardware-dependence to "GPU memory speed. ReLoRA can more effectively utilize low-bandwidth memory as it has less trainable parameters." On consumer GPUs with limited memory bandwidth, the optimizer state reduction has an outsized impact because fewer bytes need to be read/written per step, allowing the GPU to spend more time computing and less time waiting for memory transfers.

The Locally Low-Rank Training Hypothesis

The paper provides theoretical motivation via the hypothesis that neural network training is locally low-rank (Section 2, final subsection):

"For a sufficiently small trajectory, the training can be effectively approximated by a low-rank update."

This hypothesis synthesizes several lines of evidence from prior work:

• Aghajanyan et al. (2021) showed that fine-tuning updates become progressively lower-rank as model size increases, implying that the intrinsic dimensionality of learning is much smaller than the parameter count.
• Arora et al. (2019) found that stochastic gradient descent is implicitly biased toward low-rank solutions due to the dynamics of gradient flow in deep matrix factorization.
• The Lottery Ticket Hypothesis (Frankle et al., 2019) demonstrated that sparse sub-networks (which are effectively low-rank in their parameterization) can be trained to match full network performance, but only when initialized from early-training weights.

If training is locally low-rank, then within any window of a few thousand steps, the optimal weight update $\Delta W_{\text{window}}$ can be well-approximated by a rank-$r$ matrix with $r \ll \min(d_{\text{in}}, d_{\text{out}})$. ReLoRA operationalizes this: each segment between restarts trains a rank-$r$ adapter that approximates the locally optimal update for that segment. The sum across segments approximates the full training trajectory, which may be globally high-rank even though each window is locally low-rank.

The paper's empirical analysis of singular value spectra (Figure 3 and Figure 4) supports this hypothesis. Figure 3 shows that ReLoRA's total update $\Delta W$ has a singular value distribution that "exhibits a closer resemblance to full-rank training than to LoRA" — LoRA's spectrum is dominated by a few very large singular values (above 1.5) and many near-zero ones, while ReLoRA's spectrum has more mass in the intermediate range (0.1–1.0), similar to full-rank training. Figure 4 quantifies this: LoRA has many more near-zero singular values (below 0.1) than both ReLoRA and full-rank training across all projection types (Q, K, V, up, down), while ReLoRA's count of near-zero singular values is much closer to full-rank training. This provides direct evidence that the restart mechanism genuinely increases the rank of the learned update — each segment contributes information in a different subspace, and the accumulated update spans many more directions than a single low-rank adapter could.

Design Choices and Their Justifications

Why 90–99% pruning rather than 100% (complete reset)? Complete optimizer reset causes divergence (Table 6, row 3). The remaining 1–10% of unpruned optimizer state provides just enough continuity in the optimization trajectory to prevent the loss from spiking. This is a form of soft reset rather than a hard reset, analogous to how the warm start provides a soft initialization rather than starting from scratch.

Why Kaiming init for A and zeros for B at each restart? This follows the standard LoRA initialization convention. Zeros for B ensure that the adapter contributes nothing on the first step after restart — the model's output immediately after restart is identical to what it was before the restart (since $s W_A \cdot 0 = 0$). This prevents any disruption to the model's predictions during the restart transition. Kaiming init for A provides a well-conditioned starting point for gradient flow through the bottleneck, avoiding vanishing or exploding gradients in the first few steps.

Why apply ReLoRA to all linear layers, not just attention? The paper applies LoRA to "all attention and fully-connected network parameters" (Section 3). This is a design choice that maximizes the parameter reduction — feed-forward layers typically contain the majority of parameters in a transformer (the $8/3 h$ hidden size means FFN layers are larger than attention layers), so excluding them would leave substantial optimizer state overhead. The paper does not ablate which layers to wrap, but the convention from LoRA fine-tuning is that wrapping more layers generally improves performance at the cost of more trainable parameters.

Why keep embeddings and norms full-rank? These layers have relatively few parameters compared to linear layers (embeddings have $V \times d_{\text{model}}$ parameters where $V$ is vocabulary size, norms have $2 \times d_{\text{model}}$ parameters). The memory savings from low-ranking them would be small, and full-rank training of these layers may be important for maintaining representational capacity — embeddings in particular need to learn token representations from scratch during pre-training, which may require full-rank updates.

Why use fixed reset intervals rather than adaptive ones? The paper uses fixed intervals ($q$ steps between resets) rather than attempting to detect when the current subspace is exhausted (e.g., by monitoring gradient diversity or loss plateau). This is a simplicity choice: fixed intervals are easy to implement and tune, and the paper's results show that a wide range of reset frequencies (2K–5K steps) work well (Appendix A). Adaptive resetting based on the loss curve or gradient statistics is mentioned as potential future work but not explored.

Negative Result: Online ReLoRA

The paper reports an interesting negative result — "Online ReLoRA" — that attempts to increase the frequency of merging without increasing the frequency of optimizer resets (Section 4.1, Table 5). In standard ReLoRA, every merge is paired with an optimizer reset and learning rate warm-up, which limits how frequently merges can occur (too-frequent resets destabilize training). Online ReLoRA decouples these: it merges LoRA parameters very frequently (e.g., every 100 iterations) but only performs optimizer resets at the usual frequency (2K–5K iterations).

The intuition was that more frequent merges would allow the total update to achieve higher rank (more summands in Equation 2), but the results show the opposite: Online ReLoRA performs worse than standard ReLoRA at both 250M (29.31 vs. 27.66 perplexity) and 1.3B (17.80 vs. 17.36 perplexity). The paper does not deeply analyze why, but a plausible explanation is that frequent merging without optimizer reset causes the adapter to repeatedly learn in the same subspace (because the optimizer momentum hasn't been cleared), defeating the purpose of merges — the sum of many adapters all learning in the same subspace has no higher rank than a single adapter. The optimizer reset is what forces the next adapter to explore a different subspace, and decoupling merges from optimizer resets removes this forcing function.

Training Configuration Summary

For completeness, the full training configuration used across all experiments (Table 1 and Section 3):

• Architecture: Transformer with pre-normalization, RMSNorm, SwiGLU activations, $8/3 \times h$ FFN hidden size (where $h$ is the model hidden size), and rotary position embeddings.
• Dataset: C4 (Raffel et al., 2020), trained without data repetition (single epoch) on at least compute-optimal amounts of data estimated via Chinchilla scaling laws (Hoffmann et al., 2022).
• Precision: bfloat16 for all floating-point operations, FlashAttention (Dao et al., 2022) for efficient attention.
• Model sizes tested: 60M, 130M, 250M, 350M, 1.3B parameters, with the number of training tokens increasing from 1.2B (60M) to 23.1B (1.3B) following compute-optimal scaling.
• ReLoRA warm start: Initialized from a 5,000-step full-rank checkpoint, with resets every 5,000 steps thereafter, for a total of 3 resets and 20,000 total steps for most model sizes. The 1.3B model used a 10,000-step warm start with 30,000 total steps.
• Learning rates: 1e-3 (60M, 130M), 5e-4 (250M, 350M), 4e-4 (1.3B) with ReLoRA/LoRA requiring 1.5–2× larger rates.

4. Key Insights and Innovations

Innovation 1: Reframing Pre-Training as a Locally Low-Rank Optimization Problem, Not a Globally Low-Rank One

Prior to this work, the success of low-rank fine-tuning methods (LoRA, adapters, Compacter) created a natural but incorrect inference: if fine-tuning — which adapts a pre-trained model to a new task — works beautifully with low-rank updates, perhaps pre-training — which builds representations from scratch — could also be performed in a fixed low-rank subspace. The dominant assumption was that low-rank training either works or doesn't; the evaluation was binary. The paper's empirical demonstration that standard LoRA fails catastrophically at pre-training (perplexity of 47.44 vs. 33.81 for full-rank on 60M models, with the gap widening at scale) confirmed that this assumption was wrong, but more importantly, it forced the question: why does LoRA work for fine-tuning but not pre-training, and is there a way to close the gap?

The paper's answer introduces the concept of locally low-rank training — the hypothesis that neural network optimization, when viewed over sufficiently short time windows, can be well-approximated by low-rank updates, but the subspace in which those updates occur changes across the training trajectory. This is not merely an algorithmic insight; it is a fundamental reframing of what "low-rank training" means. The field had implicitly treated rank as a static property of the final weight update ($\Delta W$ from initialization to convergence). The ReLoRA paper argues that the relevant unit of analysis is the segment, not the entire trajectory.

This reframing has significant explanatory power beyond the paper's own method. It reconciles conflicting observations in the literature: Aghajanyan et al. (2021) found that fine-tuning updates become lower-rank as model size increases, Arora et al. (2019) showed SGD is biased toward low-rank solutions, and Frankle et al. (2019) demonstrated that sparse sub-networks can be trained from early checkpoints. These findings appear contradictory if one assumes a globally low-rank training process (why would lottery tickets need rewinding if the whole trajectory is low-rank?), but they cohere under the locally low-rank hypothesis: each short segment is low-rank, the segments span different subspaces, and the early part of training (the warm-start period in ReLoRA's terminology) is critical for establishing a base initialization from which subsequent low-rank segments can build. The Lottery Ticket Hypothesis with rewinding is essentially the claim that a sparse network identified early in training can serve as the first low-rank approximation, and subsequent training can be done in that reduced subspace — a special case of locally low-rank training where the rank is progressively reduced rather than maintained across restarts.

What makes this innovation conceptually distinctive is that it shifts the question from "can we train a low-rank model?" to "can we train a high-rank model using a sequence of low-rank steps?" The former is a dead end (as the paper's LoRA baselines show); the latter is what ReLoRA operationalizes. The singular value analysis in Figures 3 and 4 provides the diagnostic evidence: ReLoRA's total update $\Delta W$ exhibits a singular value spectrum with far fewer near-zero singular values than LoRA (Figure 4) and a distribution mass in the 0.1–1.0 range that "exhibits a closer resemblance to full-rank training than to LoRA" (Section 4). This is not just a performance claim — it is a mechanistic verification that the restart mechanism genuinely accumulates rank across segments, confirming the locally low-rank hypothesis empirically.

The significance of this reframing extends beyond ReLoRA itself. It suggests an entire class of training algorithms where the optimization trajectory is decomposed into segments, each of which can be performed under different constraints (low-rank, sparse, quantized) as long as the total accumulated update achieves the necessary representational capacity. It also explains why the naive approach of simply training LoRA for pre-training fails: a single low-rank subspace, spanning the entire trajectory, is fundamentally insufficient because the model needs to learn different types of representations (token embeddings, syntactic patterns, semantic relationships) that occupy different subspaces at different stages of training. The locally low-rank hypothesis is the conceptual foundation that makes parameter-efficient pre-training thinkable, and the paper's empirical validation of it — through both the performance results and the spectral analysis — is the key intellectual contribution.

Innovation 2: Identifying and Solving the Optimizer State Carryover Problem as the Primary Barrier to LoRA Restarts

The paper's diagnostic insight into why naive LoRA restarts don't work is, in itself, a significant contribution. It would have been easy — and many prior efforts implicitly did this — to assume that periodically merging LoRA adapters into the backbone and reinitializing them would naturally lead to higher-rank accumulated updates. After all, Equation 2 ($\Delta W = sW^1_A W^1_B + sW^2_A W^2_B + \cdots$) is mathematically sound, and the rank-of-sum property guarantees that the total can exceed the rank of any individual term. What the paper discovered, and what prior work missed, is that the optimizer's internal state creates an invisible coupling between adapter segments that defeats the restart mechanism.

This is a genuine diagnostic contribution rather than a simple optimization trick. The paper identified that Adam's momentum buffers ($m_t$ and $v_t$, with $\beta_1 \approx 0.9$ and $\beta_2 \approx 0.999$) retain gradient history across the restart boundary, and because these buffers encode the direction of past parameter updates, they steer the newly initialized adapter toward the same subspace as the previous one. The paper's phrasing is precise: "continuing to use old gradient moments for $W^2_A$ will guide it in the same direction as $W^1_A$ and optimize the same subspace" (Section 2). The consequence is that $W^2_A W^2_B$ learns a subspace highly overlapping with $W^1_A W^1_B$, and the total rank of the sum is only marginally higher than $r$ — exactly what Figure 4 confirms when comparing LoRA (few non-zero singular values) against ReLoRA (many more).

The ablation in Table 6 crystallizes this contribution. Row 2 (restarts added, but no optimizer reset or jagged schedule) achieves 34.25 perplexity — essentially identical to LoRA-only at 34.17. The restarts are happening mechanically, but they are not achieving their intended purpose because the optimizer state acts as a hidden persistence mechanism. The paper's solution — magnitude-based pruning of 90–99% of optimizer states at each restart — is a conceptually clean intervention that targets the root cause rather than a heuristic workaround. By erasing the accumulated momentum, the optimizer "forgets" the previous subspace and allows the new adapter to explore a genuinely different one. The finding that even 99% pruning is tolerated without performance degradation (Table 9, fine-tuning ablation) further strengthens the diagnosis: the optimizer state truly does encode persistent directional information that constrains subspace exploration, and removing it is both necessary and surprisingly non-disruptive when paired with the learning rate warm-up.

This innovation matters because it identifies a previously unrecognized interaction between parameter-efficient training and adaptive optimization. The machine learning community has largely treated optimizers as interchangeable black boxes — SGD, Adam, AdamW — without considering how their internal state dynamics interact with training methods that involve periodic parameter reinitialization. The ReLoRA paper demonstrates that this interaction is not just an implementation detail; it is the primary barrier to making restart-based training work. The concept of "optimizer state carryover" as a limiting factor for subspace exploration is novel and generalizable beyond ReLoRA: any method that involves resetting or reinitializing parameters during training (pruning-based sparse training, dynamic architecture methods, progressive network growth) must contend with this issue. The magnitude pruning solution is specific to ReLoRA, but the problem formulation is a transferable diagnostic concept.

The jagged cosine learning rate schedule (Figure 2), while mechanically simple, completes this innovation by addressing the instability introduced by optimizer state reset. The insight that dropping the learning rate to zero and re-warming prevents divergence — and that partial optimizer reset enables faster warm-up (50 steps vs. hundreds) — is a practical finding that demonstrates the interdependent nature of the restart components. Neither the optimizer reset nor the jagged schedule works alone (Table 6, rows 3 and 4: optimizer reset without jagged schedule diverges; jagged schedule without optimizer reset performs the same as LoRA). Only the combination breaks the subspace lock-in while maintaining stability. This interdependence is itself a finding: the restart mechanism is not decomposable into independent modifications but requires a coordinated intervention across the optimizer state, learning rate schedule, and adapter initialization.

Innovation 3: Empirical Demonstration That Parameter-Efficient Pre-Training Becomes More Effective with Scale — Inverting the Expected Trend

The paper's scaling analysis (Table 2, spanning 60M to 1.3B parameters) reveals a pattern that is non-obvious and practically significant: ReLoRA's efficiency gains improve with model size, not degrade. At 60M, ReLoRA achieves 34.46 perplexity vs. 33.81 for full-rank training — a gap of 0.65. At 130M, the gap is 25.04 vs. 23.65 (1.39 difference, but ReLoRA is still substantially better than the equal-parameter control at 27.30). At 250M, the gap narrows to 22.48 vs. 22.39 (0.09 difference — nearly identical). At 350M, ReLoRA reaches 19.32 vs. 18.66 (0.66 difference, but now the relative gap has shrunk substantially compared to the 60M case). At 1.3B, ReLoRA achieves 17.27 vs. 16.83 (0.44 difference), with the absolute perplexity gap smaller than at 60M despite the model being over 20× larger.

This trend is counterintuitive. One might expect that constraining a larger fraction of parameters to train in low-rank segments would impose a proportionally larger penalty on performance — the larger the model, the more parameters are "frozen" between restarts, and the more severe the representational bottleneck should be. Instead, the relative performance of ReLoRA improves with scale, tracking closer to full-rank training as model size increases. The paper does not provide an explicit theoretical explanation for this scaling behavior, but it is consistent with the broader observation from Aghajanyan et al. (2021) that larger models have lower intrinsic dimensionality in their learning dynamics. A 1.3B-parameter model may naturally explore a lower-dimensional manifold during training than a 60M-parameter model (relative to their total parameter counts), meaning that the per-segment rank constraint is less binding at larger scales.

The practical implication is significant: if this trend continues beyond 1.3B parameters — and the paper explicitly speculates that it might, stating that "the efficiency of ReLoRA increases with the model size, making it a viable candidate for multi-billion-parameter training" (Section 1) — then ReLoRA's value proposition becomes strongest precisely at the scale where full-rank training costs become prohibitive. At the 1B+ scale, where training a model from scratch requires hundreds or thousands of GPU-days, a method that saves 5.5Gb of GPU RAM and provides 9–40% wall-clock speedup while matching full-rank perplexity is not merely an academic curiosity; it is an economically meaningful optimization that could make pre-training accessible to groups with smaller compute budgets.

The paper's hardware-dependent speedup analysis (Table 7) reinforces this point. The largest relative gains (51% total speedup on 2×RTX 3090 vs. 9% on 8×A100) appear on budget hardware — exactly the kind of setup that academic labs and smaller industry teams are most likely to have access to. The attribution to "GPU memory speed" (Section 4.1) suggests that ReLoRA's optimizer state reduction reduces memory bandwidth pressure, and lower-bandwidth consumer GPUs benefit disproportionately from this. This is an important form of democratization: ReLoRA does not just improve efficiency in absolute terms; it makes efficient training more accessible on the hardware that most researchers actually have, not just the high-end clusters that large industrial labs can afford. This is a hardware-aware innovation whose significance is contextual: the 9% speedup on 8×A100 is modest, but the 51% speedup on a 2×3090 workstation could mean the difference between a project being feasible or infeasible.

Innovation 4: A Surgical Ablation Analysis That Establishes the Necessary and Insufficient Role of Each Component

The ablation study in Table 6 deserves recognition as an innovation in its own right — not for any single numerical result, but for the diagnostic framework it establishes. Most parameter-efficient training papers report an ablation comparing the full method against a baseline to show improvement. ReLoRA's ablation goes further: it systematically tests each component's contribution, reveals their interdependencies, and identifies which components are individually insufficient and which combinations are actively harmful.

The progression in Table 6 tells a clear story: (1) LoRA alone fails (34.17 perplexity), establishing the problem baseline. (2) Adding restarts without any other changes does almost nothing (34.25), establishing that mechanical restarts are insufficient when optimizer state persists. (3) Adding optimizer reset without a jagged schedule causes divergence, establishing that naive state clearing is destructive. (4) Adding a jagged schedule without optimizer reset does nothing (34.29), establishing that the learning rate adjustment alone cannot break the subspace lock. (5) Adding a warm start to LoRA (without restarts) provides the largest single improvement (25.46), establishing that initialization quality is the dominant factor. (6) The full ReLoRA recipe (all components) achieves 25.04, establishing that restarts + optimizer reset + jagged schedule provide incremental but necessary gains on top of the warm start.

This is more than an ablation — it is a failure mode analysis. Each row diagnoses a specific way the method can break: idle restarts (row 2), divergence (row 3), and ineffective scheduling (row 4). The paper does not just show that the full recipe works; it shows why each incomplete recipe fails, and those failure modes correspond to the specific mechanisms described in Section 2 (optimizer state carryover, loss instability after reset). This gives future researchers a diagnostic toolkit: if a replication attempt fails, the ablation provides a checklist for which component might be responsible.

The warm-start result (row 5) is particularly revealing. The fact that LoRA + warm start achieves 25.46 — compared to 34.17 for LoRA alone and 25.04 for full ReLoRA — demonstrates that the majority of the performance improvement comes from the warm start, not the restarts. This could be interpreted as a weakness (maybe ReLoRA is just warmed-up LoRA with a small refinement), but the paper treats it honestly: "we measured the perplexity of the warmed-up network, which equals 27.03. It outperforms all low-rank methods except for our final ReLoRA recipe but still demonstrates a significant difference from the final network." The warm start alone plateaus at 27.03; ReLoRA's restarts further reduce this to 25.04 (for the 130M model, a meaningful 2-point perplexity improvement). The contribution is incremental but real, and the ablation makes the incrementality transparent rather than hiding it behind opaque "method A vs. method B" comparisons.

This transparency is methodologically important. In a field where methods are often presented as monolithic black boxes with an "our method beats baseline" narrative, ReLoRA's ablation invites readers to understand the shape of the contribution: initialization matters most, but the restart mechanism provides a non-trivial additional gain, and that gain is only achievable through a specific combination of optimizer management and learning rate scheduling. This is a contribution to experimental methodology as much as to the specific technique.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the C4 dataset (Raffel et al., 2020), a large-scale web-crawled corpus of English text. The paper does not specify the exact C4 split, but training follows a single-epoch regime on at least compute-optimal amounts of data as estimated by the Chinchilla scaling laws (Hoffmann et al., 2022), with token counts ranging from 1.2B (60M model) to 23.1B (1.3B model) per Table 1.

• Base model(s). The paper trains transformer language models from scratch at five scales: 60M, 130M, 250M, 350M, and 1.3B parameters. The architecture closely follows LLaMA (Touvron et al., 2023), using pre-normalization, RMSNorm (Zhang and Sennrich, 2019), SwiGLU activations (Shazeer, 2020), 8/3× hidden-size FFN layers, and rotary position embeddings (Su et al., 2021). These models are chosen to span a broad range from small-scale to billion-parameter regimes where training efficiency becomes practically critical.

• Metrics. The primary metric is perplexity on the held-out validation set, reported in Tables 2, 4, and 6. Downstream evaluation uses GLUE benchmark accuracy averaged across eight tasks (Table 3). For the singular value analysis (Figures 3 and 4), the metric is the spectrum of the learned weight update $\Delta W$ — the difference between warm-start weights and final weights — specifically the distribution of singular values exceeding various thresholds, which measures the effective rank of the update.

• Baselines. Five baselines are compared:

  • Full training (Full training): Standard full-rank training of the complete model.
  • Control (Control): Full-rank training of a smaller model whose total parameter count matches the number of trainable parameters in the corresponding LoRA/ReLoRA configuration (e.g., 99M trainable for the 250M ReLoRA model).
  • LoRA (LoRA): Standard low-rank adaptation applied to all linear layers with no restarts, no warm start, trained from random initialization. This is the primary failure baseline.
  • LoRA + Warm Start (LoRA + Warm Start): LoRA initialized from a full-rank warm-start checkpoint but with no subsequent restarts, optimizer resets, or jagged scheduling.
  • ReLoRA (ReLoRA): The full proposed method including warm start, periodic restarts, optimizer state pruning, and jagged cosine scheduling.

• Generation budget / compute accounting. Compute is measured in training tokens (Table 1), with each model trained on a compute-optimal amount following Chinchilla scaling laws. Efficiency comparisons use GPU RAM consumption (Gb per GPU) and wall-clock training time (hours) at matched data volume, with throughput measured in examples per second (Table 7). The paper clarifies that warm-start cost is amortized into speedup calculations by adjusting the effective throughput: "Warm-start adjusted ReLoRA throughput" in Table 7 accounts for 33% of total training being full-rank before the switch. All comparisons use bfloat16 precision with FlashAttention (Dao et al., 2022).

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. Results are single-run measurements. The paper relies on consistency across model scales (60M through 1.3B) and across hardware configurations (Table 7) as implicit robustness evidence rather than formal statistical protocols.

Main Quantitative Results

Pre-Training Perplexity Across Model Scales (Table 2, Figure 1)

The headline result spanning five model sizes (60M through 1.3B) is that ReLoRA achieves perplexity comparable to full-rank training while training only a fraction of parameters. Across all scales, ReLoRA substantially outperforms both the Control baseline (full-rank training with matched trainable parameter count) and all LoRA variants:

• 60M: ReLoRA achieves 34.46 perplexity vs. 33.81 for full training (gap: 0.65), while LoRA alone reaches 47.44 (gap: 13.63) and the 43M-trainable Control reaches 36.52.
• 130M: ReLoRA achieves 25.04 vs. 23.65 full training (gap: 1.39), while LoRA alone reaches 34.17 (gap: 10.52) and the 72M Control reaches 27.30. LoRA + Warm Start reaches 25.46, indicating that the warm start alone closes most of the gap, and ReLoRA's restarts provide an additional 0.42 improvement.
• 250M: ReLoRA achieves 22.48 vs. 22.39 full training (gap: 0.09 — nearly identical), while the 99M Control reaches only 25.43 and LoRA alone reaches 36.60. This is the scale at which ReLoRA essentially matches full-rank performance.
• 350M: ReLoRA achieves 19.32 vs. 18.66 full training (gap: 0.66), while the 125M Control reaches 23.65 and LoRA + Warm Start reaches 19.73. The gap between ReLoRA and full training narrows relative to the 60M case despite the model being nearly 6× larger.
• 1.3B: ReLoRA achieves 17.27 vs. 16.83 full training (gap: 0.44) at 30K steps, with 250M trainable parameters out of 1.3B total. LoRA + Warm Start reaches 18.23, and the 250M Control reaches 21.73.

The pattern across scales is noteworthy: the absolute perplexity gap between ReLoRA and full training does not monotonically increase with model size despite the frozen fraction growing from ~28% (60M: 43M trainable / 60M total) to ~81% (1.3B: 250M trainable / 1.3B total). At 250M, ReLoRA essentially matches full training (gap: 0.09), and at 1.3B the gap is 0.44 — smaller in absolute terms than at 60M (gap: 0.65). This supports the paper's claim that "the efficiency of ReLoRA increases with the model size" (Section 1), though the non-monotonic behavior (gap drops to near-zero at 250M then rises to 0.44 at 1.3B) is left unexplained.

Figure 1 visualizes the 250M trajectory, showing ReLoRA (99M trainable) closely tracking full training (250M) throughout, while the pure LoRA curve (no warm start) diverges to much higher loss. The step markers at 0, 100, and 250 show the trainable parameter counts, illustrating in a single image that ReLoRA achieves parity with full-rank training using ~40% of trainable parameters at any given step.

Singular Value Spectrum Analysis: Demonstrating High-Rank Updates (Figures 3, 4, 11, 12)

The paper's central mechanistic claim — that ReLoRA genuinely learns a high-rank update through low-rank segments — is supported by singular value decomposition (SVD) of the total weight update $\Delta W$ (difference between warm-start weights at step 5,000 and final weights at step 20,000 for 350M models, and similarly for 130M models in Appendix D). The analysis compares three conditions: ReLoRA, LoRA (with warm start), and full-rank training.

Figure 3 plots the singular value spectra (frequency distributions of singular values) for four projection types — Q, K, V, and Down projections — in the 350M model. The key qualitative finding:

"While most of the singular values for LoRA are zero (Figure 4) with a noticeable number of exceptionally high values above 1.5, ReLoRA exhibits a higher distribution mass between 0.1 and 1.0, reminiscent of full-rank training."

In the K and V projections specifically, LoRA shows a concentration of singular values above 1.5 (tall, narrow peaks) with most values at or near zero, while ReLoRA's distribution spreads more evenly across the 0.1–1.0 range, visually tracking the full-rank training curve. For the Q projection, both LoRA and ReLoRA show distributions closer to full-rank, though ReLoRA more so. For the Down projection, LoRA's spectrum is sharply peaked near zero with a long tail, while ReLoRA's spectrum has substantially more mass in the 0.5–1.5 range.

Figure 4 (and Figure 12 for 130M) provides a complementary quantification: the number of singular values greater than 0.1 in the learned update for each projection type. At 350M:

• WQ: Full-rank ≈ 900, ReLoRA ≈ 750, LoRA ≈ 500
• WK: Full-rank ≈ 950, ReLoRA ≈ 600, LoRA ≈ 90
• WV: Full-rank ≈ 950, ReLoRA ≈ 650, LoRA ≈ 100
• Wup: Full-rank ≈ 1000, ReLoRA ≈ 350, LoRA ≈ 100
• Wdown: Full-rank ≈ 1000, ReLoRA ≈ 700, LoRA ≈ 50

ReLoRA consistently has substantially more non-negligible singular values than LoRA across all projection types, with the largest gaps appearing in WK, WV, and Wdown. For Wup, ReLoRA still has ~350 vs. ~100 for LoRA, but both are far from the full-rank count of ~1000 — the up-projection appears to be the hardest component for low-rank training to capture. Overall, the pattern confirms the rank-of-sum mechanism: each ReLoRA segment contributes information in a different subspace, and the total update spans many more directions than a single-rank-128 LoRA adapter could achieve.

Figure 11 (Appendix D) replicates this analysis for the 130M model with qualitatively identical findings, establishing that the spectral pattern is consistent across scales.

Scaling to 1.3B Parameters (Table 4, Figure 5)

The largest-scale experiment applies ReLoRA to a 1.3B-parameter model trained on C4, using a 10K-step warm start (33% of total 30K training steps), rank r = 128, and resets every 5K steps thereafter. Table 4 reports perplexity at three checkpoints:

ReLoRA outperforms LoRA + Warm Start at every checkpoint, and the gap widens over time: 0.56 at 15K steps, 0.74 at 20K steps, and 0.96 at 30K steps. This widening gap is the clearest evidence that the restart mechanism provides a compounding benefit — each additional restart segment adds meaningful new capacity rather than saturating. The final gap to full training is 0.44 perplexity points (17.27 vs. 16.83), representing a modest but non-trivial performance cost for the 5.5Gb RAM savings and training speedup.

Figure 5 visualizes the training trajectory, showing ReLoRA deviating from LoRA + Warm Start early and maintaining a consistent advantage throughout. The visual gap widens notably after the first restart (~15K steps), consistent with the restart mechanism adding rank.

A rank-sweep experiment on the 1.3B model (Section 4.1, "Varying ReLoRA rank") yielded an unexpected result: starting from a 5K warm-start checkpoint and training for an additional 10K steps, rank 128 achieved perplexity 19.16 while rank 512 achieved 19.00 — a difference of only 0.16. This suggests that even at 1.3B scale with hidden size 2048, the rank-128 bottleneck does not substantially constrain performance, and the benefits of ReLoRA's restart mechanism are not simply a matter of needing larger per-segment rank at larger model sizes.

Downstream Evaluation on GLUE (Table 3)

To verify that ReLoRA pre-training produces models useful for downstream tasks (not just low perplexity), the paper fine-tunes the 350M models pre-trained both full-rank and with ReLoRA on the GLUE benchmark. Table 3 reports per-task and average scores:

| Model | CoLA | STS-B | MRPC | RTE | SST2 | MNLI | QNLI | QQP | Avg | |---|---|---|---|---|---|---|---|---| | Full-rank pretrained | 35.43 | 83.85 | 76.96 | 64.26 | 88.99 | 70.98 | 83.38 | 84.49 | 73.54 | | Not pretrained | 7.59 | 22.73 | 67.00 | 51.15 | 82.61 | 60.04 | 67.92 | 78.40 | 54.68 | | ReLoRA | 31.07 | 83.33 | 78.43 | 60.65 | 89.45 | 72.27 | 83.93 | 86.01 | 73.14 |

The ReLoRA-pretrained model achieves 73.14 average GLUE score compared to 73.54 for full-rank pretraining — a difference of only 0.40 points. The per-task breakdown shows ReLoRA actually outperforming full-rank pretraining on some tasks (MRPC: 78.43 vs. 76.96; SST2: 89.45 vs. 88.99; MNLI: 72.27 vs. 70.98; QNLI: 83.93 vs. 83.38; QQP: 86.01 vs. 84.49) but notably underperforming on CoLA (31.07 vs. 35.43) and RTE (60.65 vs. 64.26). The "Not pretrained" baseline (random initialization) scores 54.68 average, confirming that both pretraining methods provide substantial benefit.

The paper notes that "the absolute values of GLUE results are expected to be quite far from state-of-the-art, because our models were pre-trained on roughly 20 times less data than T5 or BERT," which is important context — these are relatively weak pre-trained models evaluated primarily for comparative purposes, not for absolute GLUE leaderboard performance.

Training Efficiency: RAM Savings and Speedup (Section 4.1, Table 7)

The practical efficiency numbers are reported for the 1.3B model training:

• RAM: ReLoRA reduces GPU memory consumption from 27.8Gb to 22.3Gb when training with the same micro-batch size, saving 5.5Gb per GPU. This enables a 2× larger micro-batch size, improving hardware utilization.
• Total training time (8×A100): ReLoRA (including 33% warm start) took 86 wall-clock hours vs. 93.5 hours for full-rank training on the same data volume — a 9% speedup.
• Total A100-hours: ReLoRA training consumed 440 A100-hours vs. 496 A100-hours for full-rank training, saving 56 A100-hours.

Table 7 breaks down the hardware-dependence of the speedup:

The "immediate speedup" is the throughput improvement during the ReLoRA phase only (excluding warm start), while "total speedup" amortizes the 33% warm-start cost. The 51% total speedup on consumer-grade 3090s vs. only 9% on data-center A100s demonstrates that ReLoRA's efficiency benefits are most pronounced on memory-bandwidth-constrained hardware — exactly the hardware that resource-limited research groups are most likely to use.

Online ReLoRA: A Negative Result (Table 5)

The paper attempted an "Online ReLoRA" variant where LoRA parameters are merged very frequently (every ~100 steps) while optimizer resets remain at the normal frequency (2K–5K steps). The hypothesis was that more frequent merging would allow higher-rank accumulated updates. The result (Table 5) showed the opposite:

Online ReLoRA performs worse than standard ReLoRA at both scales. The paper's interpretation (implicit in the placement as a negative result) is that frequent merging without optimizer reset causes successive adapters to learn in overlapping subspaces (because the optimizer momentum persists), so the more-frequent merges do not actually increase the total rank. This negative result is informative: it demonstrates that the optimizer reset is not an implementation detail but the mechanism that forces subspace exploration, and decoupling merge frequency from reset frequency breaks this mechanism.

Ablation Studies and Robustness Checks

ReLoRA component ablation (130M model, Table 6): This is the most methodologically important ablation, testing all four ReLoRA components — restarts, optimizer reset, jagged schedule, and warm start — in isolation and combination. The baseline LoRA (no components) achieves 34.17 perplexity. Adding restarts alone (row 2) yields 34.25 — essentially no improvement, confirming that mechanical restarts without optimizer state management are ineffective. Adding optimizer reset without the jagged schedule (row 3) causes the model to diverge, establishing that naive state clearing is destructive. Adding the jagged schedule without optimizer reset (row 4) achieves 34.29 — also no improvement, confirming that learning rate adjustments alone cannot break subspace lock-in. Adding restarts, optimizer reset, and jagged schedule but no warm start (row 5) achieves 29.77 — a notable improvement over plain LoRA but still far from full-rank training (23.65). Adding the warm start to LoRA without restarts (row 6) achieves 25.46 — the single largest improvement among individual components, indicating that initialization quality dominates. The full ReLoRA recipe (row 7) achieves 25.04, providing an additional 0.42 improvement beyond LoRA + Warm Start. The full-rank training benchmark is 23.65. The key takeaway: the warm start contributes the majority of the gain, but the restart mechanism (with proper optimizer and schedule management) provides a real, measurable additional improvement.

ReLoRA rank sweep (1.3B model, Section 4.1): Testing ranks 128 and 512 starting from a 5K warm-start checkpoint and training for 10K additional steps yields 19.16 and 19.00 perplexity respectively — a difference of only 0.16. This robustness to rank suggests that even at large scale with hidden size 2048, the per-segment rank constraint is not the limiting factor; the restart mechanism's ability to accumulate rank across segments is more important than the per-segment capacity.

Optimizer pruning percentage (fine-tuning, Table 9): When fine-tuning T5-Large on QNLI, varying the optimizer state pruning rate from 85% to 99% yields QNLI accuracy ranging from 94.31 to 94.56 — essentially flat. Even 99% pruning (near-complete reset) does not harm performance, confirming robustness to this hyperparameter. This supports the paper's claim that "the model's performance remains largely unaffected even when 99% of the optimizer states are reset."

ReLoRA reset frequency (fine-tuning, Table 9): Varying the reset interval from 1000 to 6000 steps (resulting in 1 to 11 total resets) yields QNLI accuracy between 94.33 and 94.73 — also largely flat, with a slight peak at 4000 steps (2 resets) achieving 94.73. This robustness to reset frequency is practically important: the method does not require precise tuning of when to restart.

ReLoRA for fine-tuning (Appendix B, Table 8): The paper applies ReLoRA to fine-tune T5-base and T5-large on GLUE. Unlike pre-training, ReLoRA does not outperform LoRA in this setting. Average GLUE scores: T5-base ReLoRA 84.57 vs. LoRA 85.18; T5-large ReLoRA 87.47 vs. LoRA 87.99. The paper does not claim an improvement for fine-tuning, presenting this as a baseline comparison that contextualizes ReLoRA's domain of applicability — the restart mechanism is beneficial for pre-training (where the model must learn representations from scratch, requiring multiple subspaces) but unnecessary for fine-tuning (where a single low-rank subspace suffices, per Aghajanyan et al., 2021).

Warm-start duration sensitivity (Appendix E, Figure 13): Starting ReLoRA from an early 2K-step checkpoint (rather than the standard 5K) yields 23.64 perplexity for ReLoRA vs. 25.08 for LoRA + Warm Start on the 130M model — a gap of 1.44 points favoring ReLoRA. While the absolute performance is worse than with a 5K warm start (25.04), the gap between ReLoRA and LoRA + Warm Start is actually larger at this early checkpoint, demonstrating that the restart mechanism is beneficial even with suboptimal initialization. This also validates that ReLoRA's advantage is not merely an artifact of the warm-start quality.

Critical Assessment

Claim 1: "ReLoRA achieves comparable performance to regular neural network training" (Abstract, Section 4). This claim is supported with qualifications that the paper itself makes transparent. At 250M, the gap is 0.09 perplexity — essentially identical. At 1.3B, the gap is 0.44 (16.83 vs. 17.27), which is small but non-zero. At smaller scales (60M: 0.65 gap; 130M: 1.39 gap), the gap is more substantial. The claim holds most strongly at the 250M–1.3B scale where efficiency matters most, but it is not universally true across all tested scales. The paper does not overclaim here — the abstract says "comparable performance," and the numbers bear this out within a few tenths of perplexity at the largest scales. The downstream GLUE results (73.14 vs. 73.54 average) further support "comparable" rather than "identical." A missing experiment that would strengthen this claim is training a 1.3B+ model without any warm start to verify that ReLoRA does not simply converge to the warm-start initialization's performance plateau; the paper partially addresses this with the 2K warm-start experiment (Appendix E), but only at 130M.

Claim 2: "ReLoRA saves up to 5.5Gb of RAM per GPU and improves training speed by 9–40% depending on the model size and hardware setup" (Abstract, Section 4.1). Strongly supported with specific measurements. The 5.5Gb figure comes from the 1.3B model experiment comparing 27.8Gb to 22.3Gb. The 9% figure is the total speedup on 8×A100 (including warm start); the 40% figure appears in the abstract range but the highest measured total speedup is 51% (2×3090) and 32% (6×A6000), with the 40% likely referencing the 48% immediate speedup on A6000 hardware before warm-start amortization. The hardware-dependence is well-documented in Table 7. A limitation is that these measurements are for a single model size (1.3B) on specific GPU configurations; RAM savings at other scales are not reported. Additionally, the 2× larger micro-batch size enabled by RAM savings is mentioned but not separately ablated — the speedup conflates throughput gains from larger batches with any per-step compute efficiency from fewer trainable parameters.

Claim 3: "ReLoRA performs a high-rank update" (Section 4, Figures 3, 4). Supported by the SVD analysis, but with important interpretive caveats. The singular value spectra clearly show ReLoRA's update has more non-negligible singular values than LoRA's, and the distribution shape is closer to full-rank training. However, Figure 4 also shows that ReLoRA still has substantially fewer singular values above 0.1 than full-rank training for most projection types (e.g., Wup: ~350 for ReLoRA vs. ~1000 for full rank; WK: ~600 vs. ~950). The claim "performs a high-rank update" is qualitatively true relative to LoRA but quantitatively still below full-rank training's effective rank. This is consistent with the perplexity gap remaining at most scales. A missing analysis is the relationship between the number of restarts and the achieved rank — the paper uses 3 restarts in most experiments, but does not show how the effective rank grows with each additional restart, which would directly test the rank-of-sum hypothesis.

Claim 4: "The efficiency of ReLoRA increases with the model size, making it a viable candidate for multi-billion-parameter training" (Abstract, Section 1). Partially supported. The relative perplexity gap does narrow from 60M (gap: 0.65) to 250M (gap: 0.09), but widens again at 350M (0.66) and 1.3B (0.44). The trend is not monotonic, and the paper does not explain the 250M anomaly. The practical efficiency gains (speedup, RAM savings) are only measured at 1.3B, so the statement "efficiency increases with model size" cannot be verified from the reported data — it appears to be extrapolated from the observation that the trainable parameter fraction decreases with scale (43M/60M = 72% trainable at 60M vs. 250M/1.3B ≈ 19% at 1.3B) while performance remains competitive. The claim that ReLoRA is "a viable candidate for multi-billion-parameter training" is speculative; the largest tested model is 1.3B, and the dynamics at 7B or 13B are unknown. The paper is honest about this limitation, stating "Our further studies will focus on... applying it to larger models."

Weaknesses and missing experiments:

• Single training run per configuration. No error bars, no multiple seeds, no variance estimates. The singularity of each reported number makes it impossible to assess whether small gaps (e.g., 0.09 perplexity at 250M) are statistically meaningful or within run-to-run variance. This is a significant limitation given that the core claim — "comparable performance" — depends on small numerical differences.

• No experiment combining ReLoRA with the warm start on the Control baseline. The Control baseline (full-rank training of a smaller model with matched trainable parameter count) consistently underperforms ReLoRA (e.g., 25.43 vs. 22.48 at 250M). But the Control is trained from scratch, while ReLoRA benefits from the warm start. A fairer comparison would be Control trained with the same warm-start budget — does the advantage come from the restart mechanism or simply from the warm start being a better way to spend the initial compute? The ablation in Table 6 partially addresses this (LoRA + Warm Start achieves 25.46 vs. ReLoRA 25.04 at 130M), but the full-rank Control with warm start is never tested.

• No ablation on which layers are wrapped with LoRA. The paper applies LoRA to all linear layers but never tests whether wrapping only attention layers or only FFN layers would be sufficient. This is relevant because Figure 4 shows ReLoRA is much better at capturing the rank of WK, WV, and Wdown compared to Wup — perhaps different layers benefit differentially from the restart mechanism, and a selective wrapping strategy could further improve efficiency.

• C4 dataset only. All pre-training experiments use C4. The paper does not evaluate on other standard pre-training corpora (The Pile, RedPajama, etc.), so the generalizability of the locally low-rank hypothesis across data distributions is untested.

• No comparison to other low-rank training methods. Kamalakara et al. (2022), Schotthöfer et al. (2022), and Sui et al. (2023) are cited in Related Work but never implemented as baselines. While the paper argues these methods were not evaluated at scale on transformers, running at least one alternative low-rank training method at the 130M scale would substantially strengthen the claim that ReLoRA represents an advance over the state of the art rather than over an intentionally weak LoRA baseline.

• The GLUE evaluation raises a concern about the Control baseline interpretation. The ReLoRA model matches full-rank pretraining on GLUE (73.14 vs. 73.54 average) despite using only 250M trainable parameters for a 350M model. This is genuinely impressive, but the mechanisms are unclear — does ReLoRA produce better representations than full-rank training on some tasks (as the per-task scores suggest), or is this within run-to-run variance of GLUE fine-tuning? Without multiple seeds, the per-task advantages cannot be interpreted.

• Hardware speedup conflates multiple factors. The speedup numbers in Table 7 include the effect of 2× larger micro-batch sizes, which is a consequence of RAM savings but not a direct property of the algorithm (full-rank training could also use gradient accumulation with the same effective batch size, just with more communication overhead). The throughput comparison at matched micro-batch size is only reported as a RAM number (27.8Gb to 22.3Gb), not a throughput number. The paper would benefit from an ablation that isolates the throughput gain from reduced optimizer overhead vs. larger batch sizes.

• The 1.3B model results are not directly comparable to the smaller scales. The 1.3B model uses a different warm-start fraction (33% vs. 25%), different total steps (30K vs. 20K), and a learning rate of 4e-4 vs. 5e-4 for 250M/350M. These confounds make it difficult to interpret the scaling trend. The paper would benefit from a controlled scaling experiment where all ratios (warm start fraction, reset frequency relative to total steps, learning rate scaling) are held constant.

6. Limitations and Trade-offs

The Warm-Start Phase Reintroduces Substantial Full-Rank Training Cost That Is Partially Hidden by the Amortization Framework

The assumption or constraint: ReLoRA requires an initial period of standard full-rank training — called the "warm start" — before the model switches to low-rank adapter training. The paper explicitly acknowledges this in its description of the method: "we perform warm start for about 1/4 of the total training updates" (Section 4.2), and in the 1.3B experiments, the best-performing run used a 10,000-step warm start out of 30,000 total steps — a full 33% of training conducted conventionally (Section 4.1). During this warm-start phase, all parameters are trainable, consuming the same memory and compute as full-rank training, so ReLoRA's claimed efficiency gains do not apply.

The consequence: The headline speedup numbers presented in the abstract ("9-40%") are amortized figures that include the warm-start cost distributed across the entire training run. Table 7 quantifies the gap between amortized and non-amortized speedups: on 8×A100, the ReLoRA phase alone achieves 15% immediate speedup, but after warm-start amortization this drops to 9%. On 6×A6000 Ada, the gap is even more dramatic — 48% immediate drops to 32% total. On 2×RTX 3090, 102% immediate drops to 51% total. The warm start essentially creates a fixed "tax" on the total efficiency gain, and the tax rate depends entirely on the warm-start fraction. If a practitioner needs a longer warm start for a more complex dataset or larger model, the amortized efficiency gain shrinks further. Worse, the paper provides no principled method for determining the minimum warm-start duration — the choice of 1/4 to 1/3 of total steps is empirically tuned on C4 pre-training, with no evidence that it generalizes.

What evidence exists in the paper: The ablation study (Table 6) shows that removing the warm start entirely — applying LoRA with restarts, optimizer resets, and a jagged schedule but no warm start — achieves 29.77 perplexity vs. 25.04 with the warm start (130M model). The warm start contributes 4.73 perplexity points of improvement, while the full restart mechanism (beyond LoRA + Warm Start) contributes only 0.42 additional points. This reveals that the warm start is empirically the dominant contributor to ReLoRA's performance, not the restart mechanism itself. The paper states this explicitly: "we measured the perplexity of the warmed-up network, which equals 27.03. It outperforms all low-rank methods except for our final ReLoRA recipe but still demonstrates a significant difference from the final network" (Section 4.2). The Appendix E experiment (Figure 13) tests a shorter 2K-step warm start on the 130M model and finds ReLoRA still outperforms LoRA (23.64 vs. 25.08 perplexity), but with a much larger absolute gap to full-rank training, confirming that the warm-start duration directly trades off efficiency against final performance.

Mitigation status: Not addressed. The paper acknowledges the warm start as necessary (the Lottery Ticket analogy in Section 2) but treats the duration as a hyperparameter to be tuned rather than a limitation requiring a solution. The authors do not develop a cheaper alternative to the warm start (e.g., starting from a pretrained model, progressive rank increase, or learning the initialization through a low-rank warmup). The "Our further studies" statement in the Conclusion (Section 6) focuses on "improving ReLoRA performance, efficiency, applying it to larger models"— it does not explicitly target reducing or eliminating the warm-start cost. Given that the warm start contributes the majority of the performance gain over plain LoRA, this is the single most important practical limitation: ReLoRA is not a pure parameter-efficient method; it is a hybrid that pays full training costs for a substantial fraction of the run.

The Locally Low-Rank Hypothesis — and Therefore ReLoRA's Core Justification — Is Verified Only at One Training Budget Per Model Size

The assumption or constraint: ReLoRA is motivated by the hypothesis that neural network training is locally low-rank — meaning that within any sufficiently short window of a few thousand steps, the weight update can be well-approximated by a low-rank matrix, even if the total trajectory is high-rank (Section 2). The paper tests this hypothesis using a specific experimental configuration: for each model size, the total training duration is set to a compute-optimal token count via Chinchilla scaling laws (Table 1), the warm start is fixed at ~25–33% of total steps, and there are exactly 3 restarts (or fewer, as with the 1.3B model's best run using a 10K warm start and 20K additional steps of ReLoRA). The paper never varies the total training budget to test whether the locally low-rank property holds at different training durations.

The consequence: There is a fundamental ambiguity in what the paper actually demonstrates. If training were 4× longer (more tokens following Chinchilla-optimal scaling) or 4× shorter (undertraining), would the same number of restarts suffice? Would the locally low-rank hypothesis break down if segments were required to span longer intervals — i.e., is there a maximum segment length beyond which training is no longer well-approximated by a rank-128 update? Conversely, are 3 restarts sufficient, or would 10 restarts (with correspondingly shorter segments) perform even better? The paper provides no answer to these questions because the training duration, restart count, and segment length are all confounded — they are varied together as part of the experimental design rather than independently. This means a practitioner cannot determine, for a new model or dataset, how to set the reset frequency q except by replicating the paper's tuning procedure. The Appendix A recommendation that "2K iterations reset rate performed consistently well" is based on pre-training and fine-tuning experiments at the paper's specific training budgets, not a systematic sweep across training durations.

What evidence exists in the paper: The fine-tuning ablation (Table 9) varies the reset rate from 1000 to 6000 steps (resulting in 1 to 11 resets) and finds QNLI accuracy essentially flat (94.33 to 94.73), suggesting that reset frequency is not highly sensitive in the fine-tuning regime. However, fine-tuning starts from a pretrained model where the locally low-rank property is already well-established (per Aghajanyan et al., 2021), so this robustness may not transfer to pre-training from scratch. The pre-training experiments vary the warm-start duration (2K vs. 5K vs. 10K steps for 1.3B, Section 4.1) but not the number of restarts independently — the warm-start duration change also changes the effective total training budget. The rank-of-sum equation (Equation 2) predicts that performance should improve monotonically with the number of restarts (more summands means higher possible rank), but the paper never tests this prediction by sweeping the number of restarts at a fixed total compute budget. The negative result on Online ReLoRA (Table 5) — where more frequent merging with constant optimizer reset frequency degrades performance — hints that simply adding more restarts without adjusting other components is not beneficial, but the mechanism is not analyzed in depth.

Mitigation status: Partially addressed through empirical tuning. The paper provides practical guidance in Appendix A ("2K iterations reset rate performed consistently well") and notes that "we did not observe significant dependence on optimizer pruning percentage as long as it's larger than 90%." But this guidance is based on the paper's specific experimental configurations and does not constitute a principled understanding of the relationship between training duration, segment length, and required rank. The paper does not suggest future work on theoretically characterizing the locally low-rank property or developing adaptive reset schedules based on gradient diversity or subspace overlap metrics.

All Pre-Training Results Are on a Single Dataset (C4) with a Single Model Architecture Family — the Locally Low-Rank Property May Be Dataset- or Architecture-Dependent

The assumption or constraint: Every pre-training experiment in the paper uses the C4 dataset (Raffel et al., 2020) and the LLaMA-style transformer architecture (pre-normalization, RMSNorm, SwiGLU, rotary embeddings), with model sizes spanning 60M to 1.3B parameters. The paper makes no claim of generalizability to other datasets, architectures, or modalities, but it also does not explicitly acknowledge this as a limitation — the Conclusion (Section 6) states the intent to apply ReLoRA "to larger models and applying it to continued pre-training of existing large language models" without mentioning cross-domain validation.

The consequence: The locally low-rank hypothesis — that training trajectories decompose into a sequence of low-rank segments — may depend on properties of the C4 data distribution that do not hold universally. C4 is a web-crawled English text corpus that is relatively homogeneous in domain (predominantly internet text). Training on more diverse corpora (The Pile, which includes code, scientific papers, and books) or on multilingual data might require exploring broader subspaces, potentially violating the assumption that rank-128 segments suffice. Similarly, the LLaMA architecture's specific design choices — particularly SwiGLU activations and the 8/3× hidden-size FFN ratio — were designed for efficient scaling, and architectures with different properties (e.g., standard ReLU with 4× hidden size, mixture-of-experts layers, or non-transformer architectures) might exhibit different rank dynamics during training. The paper's own spectral analysis (Figure 4) reveals that the up-projection (Wup) is substantially harder for ReLoRA to capture than other projections — ReLoRA achieves only ~350 singular values above 0.1 compared to ~1000 for full-rank training on the up-projection — and a different architecture might shift this bottleneck to a different component, potentially increasing the performance gap.

What evidence exists in the paper: None. There is no cross-dataset or cross-architecture evaluation. The fine-tuning experiments (Appendix B, Table 8) use GLUE tasks rather than C4, but these start from pre-trained T5 models — they test ReLoRA as a fine-tuning method, not as a pre-training method, and therefore do not address whether the locally low-rank property holds during pre-training on non-C4 data. The Related Work section (Section 5) cites prior work on low-rank properties of transformers (Aghajanyan et al., 2021; Bhojanapalli et al., 2020; Wang et al., 2020) but none of these directly address whether the training dynamics ReLoRA exploits are architecture-invariant. The paper's observation that ReLoRA's efficiency "increases with model size" (Section 1) is based entirely on the LLaMA-style architecture trained on C4; it is unknown whether different architectures (e.g., encoder-decoder, mixture-of-experts, or non-autoregressive models) would show the same trend or the reverse.

Mitigation status: Not addressed. The paper does not identify this as a limitation and does not propose future work on cross-domain or cross-architecture validation. A practitioner considering ReLoRA for pre-training on code, scientific text, multilingual data, or with a non-LLaMA architecture would need to replicate the full experimental pipeline to determine whether the locally low-rank property holds — the paper provides no transferable guidance for this assessment.

The Singular Value Analysis Demonstrates Higher Rank than LoRA but Does Not Establish Sufficient Rank for "Comparable Performance" — and the Relationship Between Spectral Properties and Downstream Quality Is Unexamined

The assumption or constraint: The paper's mechanistic verification that ReLoRA performs a "high-rank update" relies entirely on singular value decomposition (SVD) of the learned weight update $\Delta W$ (Figures 3, 4, 11, 12). The analysis compares the number of singular values above a threshold of 0.1 across ReLoRA, LoRA, and full-rank training, showing that ReLoRA has substantially more non-negligible singular values than LoRA and that its spectral distribution is qualitatively closer to full-rank training. The paper interprets this as evidence that ReLoRA achieves high-rank training through low-rank updates, validating the rank-of-sum mechanism.

The consequence: The threshold of 0.1 is arbitrary — the paper provides no justification for this specific cutoff, and the interpretation of the SVD results depends heavily on it. More importantly, the analysis does not establish the functional significance of the spectral differences. Figure 4 shows that for most projection types, ReLoRA still has substantially fewer singular values above 0.1 than full-rank training: for Wup, approximately 350 vs. 1000; for WK, approximately 600 vs. 950; for WV, approximately 650 vs. 950. These gaps represent unrecovered rank — directions in which ReLoRA's update is effectively lower-dimensional than full-rank training. The paper does not analyze whether these missing singular values correspond to functionally important or unimportant directions in the network's representation. A model could match full-rank perplexity even with a lower-rank update if the missing singular values correspond to redundant or noise directions, or it could fail on specific downstream capabilities (e.g., rare token handling, long-range dependencies, factual recall) that depend on those directions. The GLUE evaluation (Table 3) shows that ReLoRA matches full-rank training on average (73.14 vs. 73.54) but underperforms on CoLA (31.07 vs. 35.43) and RTE (60.65 vs. 64.26) — two tasks that require linguistic acceptability judgments and textual entailment, respectively. The paper does not connect these per-task differences to the spectral gaps (e.g., whether the missing singular values in specific projection types correlate with specific downstream deficits), so the SVD analysis remains a qualitative visualization rather than a diagnostic tool.

What evidence exists in the paper: The SVD analysis in Figures 3 and 4 (and Appendix D, Figures 11 and 12) is purely descriptive — it shows that ReLoRA achieves higher rank than LoRA but does not quantify whether the achieved rank is sufficient for the task. The threshold of 0.1 appears only in Figure 4's caption and is never justified. The number of restarts (3 in most experiments) is not varied in the SVD analysis, so the relationship between restart count and accumulated rank is untested — the paper cannot say whether 4 or 5 restarts would close the spectral gap to full-rank training or whether 3 is already the point of diminishing returns. The GLUE results (Table 3) are reported at the aggregate and per-task level but are not analyzed in conjunction with the spectral properties.

Mitigation status: Not addressed. The paper treats the SVD analysis as sufficient evidence for the high-rank claim and does not acknowledge the gap between ReLoRA's spectral profile and full-rank training, the arbitrary threshold, or the lack of functional validation. The Conclusion (Section 6) states "Our further studies will focus on improving ReLoRA performance, efficiency, applying it to larger models" with no mention of developing better diagnostics for the rank-performance relationship. For a practitioner, this means the SVD analysis is interesting but not actionable — it does not help predict when ReLoRA will succeed or fail on a new task or guide how to adjust the restart schedule to achieve specific rank targets.

The Comparison Against Full-Rank Training Gives ReLoRA an Advantage by Matching Total Steps Rather than Total Compute — and the Control Baseline Is Systematically Disadvantaged

The assumption or constraint: In the main results (Table 2), ReLoRA and full-rank training are compared at the same number of training steps and same number of training tokens. However, each ReLoRA training step updates only a subset of parameters (the adapters, embeddings, and norms), while each full-rank training step updates all parameters. The computational cost per step is lower for ReLoRA — that is the entire point — but this means that at matched step counts, ReLoRA has used less total compute than full-rank training. The paper's comparison answers the question: "If you train for the same number of steps, how does ReLoRA compare?" But the more relevant question for a practitioner with a fixed compute budget is: "If you spend the same total FLOPs, how does ReLoRA compare?" The paper does not perform this FLOPs-matched comparison.

The consequence: The reported perplexity gaps (e.g., 17.27 vs. 16.83 at 1.3B) understate ReLoRA's efficiency — ReLoRA achieves these numbers with less compute per step, so the gap at matched compute would likely be smaller or potentially reversed in ReLoRA's favor. However, the paper does not quantify this, leaving the practitioner to guess whether spending the "saved" FLOPs on more training steps would allow ReLoRA to match or exceed full-rank performance. Conversely, the Control baseline — full-rank training of a smaller model with the same number of trainable parameters as ReLoRA (e.g., 99M vs. 250M total at the 250M scale) — is systematically disadvantaged: the Control has fewer total parameters and therefore lower representational capacity, but it also has the same number of trainable parameters as ReLoRA (so comparable per-step cost). ReLoRA outperforms the Control by large margins (e.g., 22.48 vs. 25.43 perplexity at 250M), but the Control is trained for the same number of steps, not the same compute budget. Since each Control step costs approximately the same as each ReLoRA step (both update ~99M parameters), a compute-matched comparison would give the Control the same number of steps and show the same result. But if the comparison question is about total parameter budget, the Control has artificially limited capacity — a fairer question would be: can we achieve ReLoRA's performance by training a medium-size model (e.g., 500M parameters) with full-rank training for the same total compute as ReLoRA + warm start? This experiment is not performed.

What evidence exists in the paper: The paper reports the per-step throughput advantage (Table 7: 15% immediate speedup on A100, 48% on A6000, 102% on 3090) and total wall-clock time advantage (9%, 32%, 51% respectively), which implicitly account for the per-step cost difference. But the core perplexity comparisons (Tables 2 and 4) are step-matched, not compute-matched, and the paper never states what the full-rank perplexity would be at matched total FLOPs (i.e., allowing full-rank training to run for proportionally more steps until the total compute budget is equalized). The Control baseline is described as "a full-rank transformer with the same number of trainable parameters as ReLoRA" (Section 3), which makes it a parameter-count control rather than a compute control. The paper does not discuss the implications of this comparison framework.

Mitigation status: Partially addressed through the speedup reporting in Table 7, which allows a reader to estimate that if full-rank training were given 9–51% more steps (depending on hardware), it might close some of the perplexity gap. But this requires extrapolation — the full-rank scaling curve at extended step counts is unknown, and the paper provides no such extrapolation. A compute-matched perplexity comparison (where full-rank training receives proportionally more steps equal to the speedup factor) would directly answer the practitioner's question and is a notable omission.

ReLoRA Provides No Efficiency Benefit — and May Impose Overhead — During Inference, Limiting Its Value to the Training Phase Only

The assumption or constraint: ReLoRA is exclusively a training-time method. After the final merge-and-reinit, the adapters are absorbed into the backbone weights, and the resulting model is "a standard dense network with no adapter overhead — the $W^i$ matrices now equal the original warm-start weights plus the sum of all adapter contributions" (as described in Algorithm 1). The paper emphasizes this as a feature: the model at inference time is architecturally identical to a full-rank trained model, requiring no special handling. However, this also means that all of ReLoRA's efficiency gains accrue during training only. The model at inference has the full 1.3B parameters (or whatever the total model size is), consuming exactly the same memory and compute as a conventionally trained model of the same architecture.

The consequence: ReLoRA helps with the supply-side problem of training large models (making pre-training cheaper and more accessible) but does nothing for the demand-side problem of serving them. In deployment scenarios where inference costs dominate — high-traffic APIs, on-device inference, batched evaluation — ReLoRA provides no advantage over standard training. Moreover, ReLoRA's training process produces a model that is functionally identical in architecture to a full-rank trained model, so any downstream compression (quantization, pruning, distillation) must be applied post-hoc, exactly as with a conventionally trained model. This is in contrast to methods that train inherently sparse or low-rank models that maintain their efficiency at inference time (e.g., lottery ticket pruning applied during training, or structured sparsity methods). The paper's framing — "parameter-efficient pre-training" — is accurate but potentially misleading: the efficiency is in the training process, not the resulting model.

Additionally, the per-step training speedup of ReLoRA partially comes from the reduced optimizer state and communication overhead (Section 2, "Enhancing computational efficiency"), but the adapter forward pass adds computational overhead relative to a model that was simply smaller: computing $y = xW + s \cdot xW_A W_B$ requires the base forward pass $xW$ (which must be done anyway) plus the additional low-rank computation $xW_A W_B$. This extra computation is small (proportional to $r \times (d_{\text{in}} + d_{\text{out}})$ rather than $d_{\text{in}} \times d_{\text{out}})$), but it is non-zero. The paper does not break down the per-step compute into its components, so it is unclear what fraction of the throughput advantage comes from reduced optimizer/communication overhead vs. what fraction is consumed by the adapter forward pass overhead. On hardware with abundant compute but limited memory bandwidth (like consumer GPUs), the optimizer state reduction likely dominates; on hardware with abundant bandwidth but limited compute, the adapter overhead might partially offset the gains.

What evidence exists in the paper: The throughput measurements in Table 7 are end-to-end examples-per-second numbers that include all overhead, so the net speedup is accurately reported. However, the paper does not decompose this into computation vs. communication vs. memory components, making it difficult to predict ReLoRA's efficiency on hardware configurations not tested. The paper mentions that ReLoRA "reduces the bandwidth requirements in distributed setups, which are often the limiting factor in large-scale training" (Section 2), suggesting the authors believe communication reduction is the primary driver, but this claim is not empirically isolated (e.g., via a single-GPU throughput comparison that eliminates communication).

Mitigation status: Not addressed as a limitation. The paper's framing presents the inference-time architectural equivalence as a feature — and for many use cases, it is (simplicity of deployment, no custom inference kernels needed). But the paper never discusses the tradeoff: training efficiency at the cost of no inference efficiency improvement. The Conclusion (Section 6) does not mention inference efficiency or post-training compression as areas for future work. A practitioner building a system where both training and inference costs matter would need to factor in that ReLoRA reduces only the training portion of the total cost of ownership, and the inference portion is unchanged.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a fundamentally new framing for parameter-efficient training: locally low-rank optimization, where the training trajectory is decomposed into a sequence of short segments, each well-approximated by a low-rank update but spanning different subspaces. Prior to ReLoRA, the field treated "low-rank training" as a binary property — either a model is low-rank (trained with fixed low-dimensional constraints) or it is not (trained full-rank). The failures of standard LoRA applied to pre-training from scratch (perplexity of 47.44 vs. 33.81 at 60M scale, Table 2) were interpretable as evidence that pre-training requires high-rank updates — end of story. ReLoRA breaks this binary by demonstrating that the total update matters, not the per-step update, and that a high-rank network can be built from low-rank components if those components are allowed to explore different subspaces across training. This reframe shifts attention from whether training is low-rank to at what temporal granularity training exhibits low-rank structure — a question the field had not previously operationalized.

The magnitude of this shift is best characterized as a reframing with empirical teeth, not a paradigm shift. The deep conceptual machinery — rank of sum, the lotteryticket hypothesis with rewinding, the intuition that SGD is biased toward low-rank solutions (Arora et al., 2019) — already existed in the literature. ReLoRA's contribution is synthesizing these observations into a testable claim (training is locally low-rank) and building a working algorithm that validates it. The paper does not overturn any established theory; what it overturns is the tacit assumption that training from scratch requires all parameters to be trainable at every step. The demonstration that 250M trainable parameters can match 1.3B-parameter full-rank training at the 250M scale (Table 2: 22.48 vs. 22.39 perplexity) is a concrete existence proof that the assumption was wrong.

The paper also reconciles a latent tension in the literature between scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), which imply that bigger models perform better and therefore must be trained, and the Lottery Ticket Hypothesis (Frankle et al., 2019), which implies that most of those parameters are redundant. These two bodies of work appear contradictory: if lottery tickets exist, why does scaling work? ReLoRA's locally low-rank hypothesis offers a synthesis: the lottery ticket corresponds to the warm-start initialization — the early phase of dense training that establishes a good starting point — and the subsequent training is a sequence of low-rank refinements, each exploring a different sparse subspace. The full model's capacity is the union of these subspaces, not any individual one. This explains why pruning at initialization fails (the lottery ticket needs the warm start) while pruning after training succeeds (the full model has explored many subspaces and the pruned version preserves the essential ones). ReLoRA does not prove this synthesis — the paper does not test lottery ticket extraction on ReLoRA-trained models — but it provides the conceptual language for formulating the hypothesis.

A practical consequence of this reframing is that optimizer state management becomes a first-class design consideration in parameter-efficient training. Prior work on low-rank training (Schotthöfer et al., 2022; Sui et al., 2023; Kamalakara et al., 2022) treated the optimizer as an interchangeable black box. ReLoRA's ablation (Table 6) demonstrates that the optimizer's internal momentum buffers are the primary barrier to making restart-based training work — those buffers encode gradient history that locks new adapters into old subspaces regardless of parameter reinitialization. The solution (magnitude pruning of 90–99% of optimizer states) is specific to ReLoRA, but the diagnostic concept — optimizer state carryover as a subspace lock-in mechanism — is a general insight applicable to any training method that involves periodic parameter resetting (dynamic sparsity, progressive network growth, continual learning with parameter expansion). This insight reframes optimizer design for non-stationary training regimes: the optimizer's memory is not just a convergence accelerator but also a subspace constraint, and controlling that memory becomes a design parameter when the goal is to explore diverse parameter subspaces.

The paper also shifts the research emphasis for parameter-efficient pre-training away from compressing individual steps and toward managing the trajectory. Methods that attempt to make each update low-rank by decomposing weight matrices once and training the factors (the naive LoRA-for-pretraining approach) are shown to be fundamentally limited. The productive direction is methods that allow the low-rank subspaces to change over time — restarts, mixture of adapters, or adaptive rank allocation. This insight redirects attention from linear algebra (finding better matrix decompositions) to optimization dynamics (understanding when and how the gradient subspace changes during training). The paper's negative result on Online ReLoRA (Table 5: more frequent merging without corresponding optimizer resets degrades performance) reinforces this: it is not enough to mechanically accumulate low-rank updates; the optimizer must be forced to explore new directions.

What becomes less attractive after this paper is the simple "try LoRA for pre-training and see if it works" approach. The paper establishes conclusively that it does not work at scale (Table 2: LoRA alone achieves 57.11 perplexity at 350M vs. 18.66 for full training), and the failure mode is now understood — a single low-rank subspace is insufficient for the diverse representational demands of learning from scratch. Researchers interested in efficient pre-training can skip this dead end and focus directly on restart-based or adaptive-rank methods. Similarly, the paper makes pure "low-rank network" approaches (training a permanently low-rank model, as in the Control baselines of Tables 2 and 4) less attractive: the Control consistently underperforms even LoRA + Warm Start, let alone ReLoRA, confirming that the issue is not just parameter count but the distribution of representational capacity across the training trajectory.

Follow-Up Research This Work Enables

Scaling ReLoRA to 7B+ parameters to test whether the efficiency trend continues. The paper's central extrapolative claim — "the efficiency of ReLoRA increases with the model size, making it a viable candidate for multi-billion-parameter training" (Section 1) — is supported by a trend across 60M–1.3B parameters, but the largest tested model is 1.3B. At 7B or 13B parameters, three things could happen: the performance gap to full-rank training could continue to narrow (as the locally low-rank hypothesis would predict, since larger models have lower intrinsic dimensionality per Aghajanyan et al., 2021), it could stabilize, or it could start to widen again if the fixed rank-128 per-segment constraint becomes a bottleneck despite restarts. A scaling law experiment training LLaMA-style models at 1.3B, 2.7B, 6.7B, and 13B parameters with ReLoRA and full-rank training — all with matched data (C4 or The Pile) and matched Chinchilla-optimal token counts — would directly test whether the gap at 1.3B (0.44 perplexity, Table 4) drops toward zero or diverges. The experiment should also sweep the per-segment rank (128, 256, 512, 1024) at each scale to determine whether the optimal rank scales with hidden size, since the paper's single rank-128 vs. 512 comparison at 1.3B (19.16 vs. 19.00 perplexity) found minimal difference but this may not hold at larger hidden dimensions where the rank fraction $r / d_{\text{model}}$ becomes very small. If the trend continues, this would establish ReLoRA as a genuinely scalable alternative to full-rank training at industrially relevant model sizes.

A FLOPs-matched comparison where full-rank training receives proportionally more steps to equalize total compute. The paper's core comparisons (Tables 2 and 4) match models at equal step counts. Since each ReLoRA step updates fewer parameters, ReLoRA uses less compute per step, meaning step-matched comparisons understate ReLoRA's efficiency. A FLOPs-matched experiment would give full-rank training proportionally more steps — e.g., if ReLoRA achieves 15% immediate speedup on A100 (Table 7), full-rank training gets 15% more steps — and compare final perplexity. This would answer the practitioner's question: "Given a fixed compute budget, which training method produces the better model?" The experiment should be done at multiple scales (250M, 1.3B) and with the warm-start cost properly amortized into the total FLOP count. Depending on the outcome, this could strengthen ReLoRA's case (if ReLoRA matches or beats full-rank at equal FLOPs) or reveal that ReLoRA's step-matched advantage partially comes from using less total compute (if full-rank closes or reverses the gap when given equal FLOPs). The Control baseline should also receive the same FLOP-matched treatment — training a smaller model for proportionally more steps until the total compute matches ReLoRA + warm start. This would clarify whether ReLoRA's advantage over the Control is an artifact of step-count matching or a genuine efficiency improvement.

Testing the locally low-rank hypothesis with adaptive reset scheduling based on subspace overlap. ReLoRA uses fixed reset intervals (every 2K–5K steps). The locally low-rank hypothesis predicts that resets should ideally occur when the current low-rank subspace stops being a good approximation — when the gradient direction shifts to a substantially different subspace. This can be operationalized: at each step, compute the principal subspace angle or Grassmann distance between the current LoRA adapter's column space and the column space from $k$ steps ago, and trigger a reset when the overlap drops below a threshold. An adaptive scheduler would test whether fixed-interval resets are unnecessarily conservative (resetting before the subspace changes) or insufficiently responsive (not resetting fast enough when the gradient shifts rapidly). The experiment would compare fixed-interval ReLoRA against adaptive ReLoRA on a pre-training run, measuring both final perplexity and the number of resets triggered. If adaptive resetting achieves equal or better perplexity with fewer resets, it would validate the locally low-rank hypothesis mechanistically and produce a hyperparameter-free version of ReLoRA. A particularly informative variant would be to track subspace overlap separately for different layer types (attention projections vs. FFN projections) since Figure 4 shows they have different rank dynamics — perhaps attention layers need more frequent resets than FFN layers, and layer-specific adaptive scheduling could further improve efficiency.

Applying ReLoRA to continued pre-training of existing large models and measuring knowledge retention. The paper demonstrates ReLoRA for pre-training from scratch, but an equally important use case is continued pre-training — taking an existing pre-trained model (e.g., LLaMA-2 7B) and continuing training on new data (domain adaptation, temporal updates, multilingual expansion) using ReLoRA. This is a higher-stakes test because the warm start comes "for free" from the existing pre-trained weights, and the goal is to learn new knowledge without catastrophic forgetting of old knowledge. The locally low-rank hypothesis would predict that continued pre-training is well-suited to ReLoRA: the model already has high-quality representations, and the updates needed to incorporate new data should be low-rank per segment. The experiment would compare ReLoRA-based continued pre-training against (a) full-rank continued training (if feasible), (b) LoRA-based continued training without restarts, and (c) experience replay or elastic weight consolidation baselines. Metrics would include perplexity on the new domain, perplexity on the original domain (catastrophic forgetting), and downstream task performance on benchmarks from both domains. The paper's existing GLUE results (Table 3) suggest ReLoRA-pretrained models perform similarly to full-rank pretrained models on downstream tasks, but this experiment would test whether ReLoRA can update an existing model without degrading its existing capabilities — a problem where parameter-efficient methods are already preferred (LoRA for fine-tuning) but where the restarts mechanism might provide additional benefits by allowing the model to explore different subspaces for old vs. new knowledge.

ReLoRA for non-transformer architectures and non-text modalities to probe the generality of locally low-rank training. The paper's experiments are exclusively on LLaMA-style transformer language models trained on text. The locally low-rank hypothesis is stated in general terms ("neural network training is locally low-rank," Section 2), but it has only been tested in one architectural and data context. Applying ReLoRA to (a) vision transformers (ViT) trained on ImageNet, (b) convolutional networks (e.g., ConvNeXt) trained on image classification, (c) encoder-decoder transformers (T5-style) for text-to-text tasks, and (d) diffusion models for image generation would reveal whether the periodic-restart mechanism is a universal property of neural network optimization or specific to autoregressive language modeling. Each modality has different gradient dynamics — CNNs have spatial parameter sharing, diffusion models have denoising objectives with different loss landscapes — and the locally low-rank hypothesis may or may not hold. A particularly strong negative result — e.g., ReLoRA failing on a ConvNeXt trained from scratch despite matching transformer perplexity — would be scientifically valuable because it would delineate the boundary conditions. A positive result on, say, diffusion models would be practically impactful given the enormous training costs of state-of-the-art image generators. Each experiment should include the full spectral analysis (as in Figures 3 and 4) to characterize the rank dynamics of the total update, plus an ablation of warm-start duration to determine whether the warm-start dependency is universal or architecture-specific.

Understanding why the up-projection (Wup) resists low-rank approximation and developing targeted interventions. Figure 4 reveals a striking asymmetry: ReLoRA captures ~65% of full-rank training's effective rank for most projections (Q, K, V, Wdown) but only ~35% for the up-projection (Wup). The paper notes this pattern in the spectral analysis but provides no explanation or follow-up. This is a concrete, well-defined research question: what property of the SwiGLU up-projection makes its training dynamics higher-rank (or less locally low-rank) than other linear layers? Possible explanations include: the SwiGLU gating mechanism (gate(x) * up(x)) creates multiplicative interactions that induce higher-rank gradient structure; the up-projection has a larger output dimension (8/3 × h for the FFN hidden size vs. h for attention projections), making the intrinsic dimensionality proportionally higher; or the up-projection learns different types of features (e.g., token-specific vs. positional) that require spanning more directions. A diagnostic experiment would isolate these hypotheses: (a) replace SwiGLU with standard ReLU-GELU and measure whether the Wup gap narrows, (b) vary the FFN expansion ratio (4× vs. 8/3×) to test the dimensionality hypothesis, and (c) apply a larger per-segment rank specifically to Wup (e.g., rank 256 for Wup, rank 128 for everything else) and measure whether this closes the perplexity gap to full-rank training without substantially increasing memory. If targeted rank allocation works, it would refine ReLoRA into a layer-adaptive method where different layers get different rank budgets based on their spectral properties — a natural extension that the paper's own data motivates but does not pursue.

Practical Applications and Downstream Use Cases

Pre-training large language models on academic or small-industry GPU clusters. The paper's hardware-dependent speedup analysis (Table 7) shows that ReLoRA's efficiency gains are largest on consumer and prosumer GPUs: 51% total speedup on 2×RTX 3090 (a ~$3,000 setup) vs. only 9$80,000 setup). This directly enables pre-training experiments that would otherwise be infeasible. A research group with a 6×A6000 Ada workstation (roughly $30,000–40,000 in GPUs) can train a 1.3B-parameter model in 119 wall-clock hours using ReLoRA vs. 152 hours with full-rank training — saving 33 hours (21%) per training run. At the typical pace of academic research, where multiple training runs (hyperparameter sweeps, ablations, architectural variants) are needed, this 21% reduction compounds to saving days or weeks of total compute time. More importantly, the 5.5Gb RAM savings per GPU (from 27.8Gb to 22.3Gb for the 1.3B model) means that models which would not fit in GPU memory at all with full-rank training (requiring model parallelism or gradient checkpointing that further slows training) become trainable on the same hardware with ReLoRA. A group with 24Gb GPUs (RTX 3090/4090-class) training a 1.3B model would struggle with full-rank training's 27.8Gb requirement (forcing aggressive gradient checkpointing or offloading) but could train comfortably with ReLoRA's 22.3Gb. This is a concrete accessibility improvement: ReLoRA shifts the hardware threshold for training billion-parameter models downward by roughly one GPU tier.

Continued pre-training of open-source models for domain adaptation. Organizations that deploy open-source LLMs (e.g., LLaMA-2 7B, Mistral 7B) for specialized domains — legal, medical, financial, technical documentation — routinely continue pre-training on domain-specific corpora to improve in-domain performance. Full-rank continued training of a 7B model is expensive (requires at least 4–8 GPUs with high memory) and risks catastrophic forgetting of general capabilities. Standard LoRA is already widely used for this purpose but is limited to a single low-rank subspace across the entire adaptation. ReLoRA's restart mechanism offers a middle ground: the warm start is "free" (the existing pre-trained model), the low-rank segments keep memory requirements manageable (consistent with existing LoRA setups), but the periodic restarts allow the model to explore different subspaces for different types of domain knowledge (e.g., legal terminology in one segment, case law reasoning patterns in another). The paper's finding that restarts provide measurable improvement over LoRA + Warm Start (0.42 perplexity at 130M, Table 6; widening gap at 1.3B in Figure 5) suggests that for a given continued training budget, ReLoRA would extract more value than standard LoRA. Practitioners would set the warm start to zero steps (starting from the pre-trained weights), set the reset frequency based on the adaptation corpus size (e.g., every 1,000–2,000 steps for a small corpus, mirroring the paper's 2K–5K recommendation from Appendix A), and train with the same memory footprint as LoRA. The risk of catastrophic forgetting could be monitored by periodically evaluating on general benchmarks (MMLU, HellaSwag) during continued training — the paper's GLUE results (Table 3, ReLoRA matches full-rank pretraining at 73.14 vs. 73.54 average) provide indirect evidence that ReLoRA-trained models retain general capabilities, but this needs direct evaluation in a continued-pretraining context.

Efficient pre-training for multilingual or multi-domain models with constrained total compute. Organizations building models that must cover many languages or domains face a resource allocation problem: a fixed training budget must be distributed across diverse data sources. ReLoRA's segment-based structure suggests a natural extension: after a shared full-rank warm start on a balanced multilingual/multi-domain corpus, different ReLoRA segments could be trained on different data mixtures (e.g., segment 1 focuses on high-resource languages, segment 2 on low-resource languages, segment 3 on code). Because each segment learns in a different low-rank subspace, the model could potentially allocate distinct representational capacity to different domains without interference. The paper's evidence that ReLoRA segments learn in different subspaces (Figures 3 and 4: the spectral gap between ReLoRA and LoRA indicates subspace diversity) provides the mechanistic basis for this hypothesis, though the paper never tests multi-domain training. A practitioner could implement this by varying the data sampling weights across ReLoRA segments — a training run where the first adapter segment uses uniform domain sampling, the second oversamples low-resource domains, and the third returns to uniform, with the hypothesis that each segment's adapter specializes in different aspects of the data distribution. The 5.5Gb RAM savings per GPU (27.8Gb to 22.3Gb) would allow this multi-segment training to use larger per-GPU batch sizes, which is particularly valuable for low-resource domains where large batches of in-domain data may be hard to construct without excessive padding or repetition.

Training data curation and model development in compute-constrained research environments. The paper's finding that a 250M-parameter model trained with ReLoRA (99M trainable parameters) achieves essentially the same perplexity as a 250M full-rank trained model (22.48 vs. 22.39, Table 2) has direct implications for the model development workflow. In typical LLM research, architectural innovations, training recipe improvements, and data curation strategies are tested at small scale (hundreds of millions of parameters) before being applied to production-scale models. ReLoRA allows these small-scale experiments to be more representative of large-scale behavior: a 250M ReLoRA model with 99M trainable parameters more closely approximates the dynamic range of a full 250M model than would a 99M full-rank Control model (which achieves only 25.43 perplexity — a 3-point gap vs. 0.09 for ReLoRA). This means researchers can iterate on training recipes at small scale with ReLoRA and have greater confidence that the results will transfer when scaled up. The RAM savings (5.5Gb per GPU at 1.3B, presumably proportionally lower but still significant at 250M) and speedup (presumably following the hardware-dependent pattern in Table 7) mean that a researcher using a single RTX 3090 can run more experiments per day, accelerating the research cycle. Concretely, if a full-rank 250M training run takes 10 hours on a 3090, and ReLoRA provides roughly 50% speedup (extrapolating from Table 7's 51% for 1.3B), the same experiment takes ~7 hours — fitting into an overnight run rather than spanning into the next workday, enabling daily experimentation cycles.

When to Prefer This Method

The paper does not articulate a systematic tradeoff framework against named alternatives beyond comparing ReLoRA to full-rank training and to standard LoRA. The control flow is implicit in the results: ReLoRA is preferable when (a) pre-training from scratch with limited compute, (b) the model size is large enough that the optimizer state memory reduction matters (roughly 250M+ parameters, where the trainable/frozen parameter ratio becomes favorable), and (c) the user is willing to pay the warm-start cost (25–33% of total steps at full cost) in exchange for 9–51% speedup on the remaining steps. ReLoRA is not preferable for fine-tuning (Table 8: ReLoRA underperforms LoRA on GLUE, 84.57 vs. 85.18 for T5-base), for inference-time efficiency (the final model is a standard dense network), or for very small models (at 60M, the gap to full-rank training of 0.65 perplexity is proportionally larger than at 250M+). These boundaries emerge from the data but are not formalized as decision rules by the authors, so a structured tradeoff matrix is not provided here — the paper's contribution is an existence proof and a diagnostic toolkit, not a deployment playbook with explicit alternatives.