VeRA: Vector-based Random Matrix Adaptation

🎯 Pitch

VeRA slashes trainable parameters for fine-tuning by 10–100× versus LoRA while matching performance—simply by freezing a single pair of random matrices shared across all layers and learning tiny scaling vectors instead. This means per-user or per-task model adaptations can be regenerated from a single random seed, eliminating the massive storage bottleneck for personalized LLM deployments.

1. Executive Summary

This paper introduces Vector-based Random Matrix Adaptation (VeRA), a parameter-efficient finetuning method that reparametrizes the low-rank weight updates of LoRA by freezing a single pair of randomly initialized matrices shared across all adapted layers and learning only small trainable scaling vectors instead (one vector modulates rows/columns of the frozen matrices, the other scales the final output). Evaluated on the GLUE benchmark with RoBERTa models, the E2E benchmark with GPT-2, instruction-tuning of Llama 7B and 13B, and image classification with Vision Transformers, VeRA achieves comparable or better performance than LoRA while using an order of magnitude fewer trainable parameters—a tenfold reduction on GLUE for RoBERTa-large (61K vs. 800K parameters at matched accuracy) and a 100× reduction on Llama 13B instruction-tuning (2.4M vs. 250.3M parameters while scoring 5.22 vs. 5.31 on MT-Bench). The method's parameter count scales as L_tuned × (d_model + r) rather than LoRA's 2 × L_tuned × d_model × r, establishing that effective adaptation can be achieved through learned scaling of frozen random projections rather than through learned low-rank updates, with the primary practical benefit being that the frozen matrices can be regenerated from a random seed rather than stored, substantially reducing the memory footprint for deployments requiring numerous per-user or per-task adapted models.

2. Context and Motivation

The Storage Challenge in Model Personalization

The paper addresses a specific, acute bottleneck in the deployment of large language models: the storage cost of maintaining many adapted versions of a single base model. This problem emerges naturally in two common deployment scenarios:

Multi-user personalization. Consider a cloud-based operating system assistant that continuously adapts to individual user behaviors, preferences, and feedback patterns. Each user requires a separately finetuned checkpoint. With millions of users, the aggregate storage requirement for these adapted weights becomes prohibitive even if each individual adaptation is modest.

Multi-task serving. A single model serving many downstream tasks (sentiment analysis, summarization, question answering, code generation) benefits from having task-specific adaptations loaded on demand. Latency-sensitive serving systems need these adapted weights in GPU memory, but GPU memory is scarce relative to the number of potential tasks.

The paper makes this concrete with a quantitative example in Section 1: applying LoRA with rank 16 to the query and value projection layers of GPT-3 requires at least 288 MB of storage per adapted model in single-precision. At one million finetuned weights — corresponding to, say, one million users — this balloons to 275 TB. The storage infrastructure costs for such a deployment dwarf the computational costs of the finetuning itself.

This problem is not hypothetical. The paper explicitly frames it within the context of "the recent proliferation of language models and their deployment in personalized assistants, edge devices, and similar applications," arguing that storage efficiency — not just training efficiency — is becoming the dominant constraint.

Why Existing Parameter-Efficient Methods Are Insufficient

The paper positions itself against the backdrop of a rich line of parameter-efficient finetuning (PEFT) methods, each of which reduces the number of trainable parameters compared to full finetuning, but none of which drive the parameter count low enough to solve the storage problem at scale.

Adapter-based methods (Houlsby et al., 2019; Lin et al., 2020; Pfeiffer et al., 2021; Rücklé et al., 2021) insert small trainable bottleneck layers between the frozen pretrained modules. While adapters reduce trainable parameters versus full finetuning, they have two drawbacks: they introduce additional inference latency (since the adapter layers sit in the forward pass and cannot be merged into the frozen weights), and they still require storing the adapter weights for every adapted model — typically hundreds of thousands to millions of parameters per task.

BitFit (Zaken et al., 2022) takes an extreme approach, finetuning only the bias vectors while keeping all weight matrices frozen. This reduces trainable parameters dramatically (to ~0.1M for RoBERTa-base, as shown in Table 2), but the paper's results show it underperforms LoRA on the GLUE benchmark (85.4 average vs. 86.6 for LoRA on RoBERTa-base), suggesting that bias-only adaptation sacrifices too much expressive capacity.

LoRA (Hu et al., 2022) represents the state-of-the-art that the paper directly challenges. LoRA's key insight is that the weight update ΔW during finetuning can be constrained to a low-rank decomposition ΔW = BA, where B ∈ ℝ^(m×r) and A ∈ ℝ^(r×n) with r ≪ min(m, n). This reduces trainable parameters from m × n to r × (m + n). Critically, LoRA incurs no inference latency because the trained low-rank matrices can be merged into the frozen pretrained weights: W = W₀ + BA.

However, the paper identifies a gap between LoRA's achieved parameter reduction and what might be theoretically possible. Citing Aghajanyan et al. (2021), the paper notes:

"the upper bound for intrinsic dimensions is much smaller than what is typically utilized in such methods. For instance, the d₉₀ for RoBERTa-base is reported to be 896, whereas authors of the LoRA paper reported using 0.3M trainable parameters for this model, suggesting that the parameter count could be reduced further."

Here, d₉₀ is the intrinsic dimension — the smallest subspace dimension that achieves 90% of the full training metric, as defined by Li et al. (2018). The gap between 896 and 300,000 suggests there is substantial room for further compression. LoRA's parameter count grows as 2 × L_tuned × d_model × r, meaning it scales linearly with model width. For large models like GPT-3 (d_model = 12,288), even rank-1 LoRA requires 4.7M parameters (Table 1). This is the gap VeRA targets.

AdaLoRA (Zhang et al., 2023b) attempts to address parameter efficiency within the LoRA framework by dynamically pruning less important components of the low-rank matrices and reallocating the parameter budget to more critical layers. The paper acknowledges this as a step in the right direction — "dynamically allocating parameters to more critical layers" — but argues that a fundamentally different parameterization can achieve substantially greater reductions, "tolerating a marginal performance degradation."

The Theoretical Foundation: Random Projections and Intrinsic Dimensionality

The paper's approach is not developed in a vacuum. It draws on two converging lines of evidence that together suggest frozen random matrices with learned scaling might be sufficient for adaptation:

Low intrinsic dimensionality of finetuning. Aghajanyan et al. (2021) demonstrated that the effective parameter space for finetuning pretrained models is much smaller than the full parameter count — often just hundreds of dimensions. Training only a small number of parameters, randomly projected back into the full parameter space, could recover 90% of full finetuning performance. This result is the foundation of the paper's intuition: if the intrinsic dimension is small, then learning to modulate a fixed random basis (rather than learning the basis itself) might suffice.

Surprising effectiveness of random weights. Multiple independent research threads have shown that neural networks with frozen, randomly initialized weights can perform surprisingly well when small subsets are finetuned or when only normalization parameters are trained (Ramanujan et al., 2020; Lu et al., 2022; Schrimpf et al., 2021; Frankle et al., 2021). The paper cites these results as evidence that "frozen, randomly initialized models, with small sections finetuned, can perform surprisingly well." In the context of finetuning adaptation, this suggests that the random basis provided by frozen matrices might already span a useful subspace — the learning task reduces to selecting and scaling the right directions within that subspace rather than discovering the directions themselves.

The paper explicitly connects these two threads in Section 2:

"Collectively, these works create a compelling case for the utilization of frozen random matrices in finetuning methods, providing both a theoretical and an empirical foundation for the approach taken in this paper."

A particularly close precedent is the work of Ruiz et al. (2023), which used frozen random matrices inside LoRA for personalization of text-to-image models. VeRA can be seen as generalizing this idea — using shared frozen matrices across all layers rather than per-layer frozen matrices — and demonstrating its effectiveness across language understanding, language generation, instruction following, and image classification tasks.

Positioning: VeRA as a Reparameterization, Not a Constraint

The paper carefully distinguishes its approach from the conceptual framework of LoRA. LoRA constrains weight updates to be low-rank: ΔW must live in the subspace spanned by the learned matrices A and B. This is an inductive bias — the learning algorithm is explicitly told to find solutions in a low-dimensional manifold.

VeRA takes a different conceptual approach. The frozen matrices B (ℝ^(m×r)) and A (ℝ^(r×n)) are not required to be low-rank in the traditional sense because r can be large (the paper uses r = 1024 for RoBERTa-base and r = 256 for RoBERTa-large on GLUE; r = 1024 for instruction-tuning). These matrices are frozen and shared, so their storage cost is zero (regenerated from a seed). The trainable parameters are the diagonal scaling vectors d ∈ ℝ^(1×r) and b ∈ ℝ^(m×1), which modulate rows and columns of these frozen matrices.

This is a fundamentally different parameterization: rather than learning what the basis directions should be, VeRA learns only how to scale a fixed, randomly chosen basis. The expressivity comes from the fact that r can be large without incurring storage costs, since only the scaling vectors (not the matrices themselves) are stored. As the paper notes in Section 3.1:

"B ∈ ℝ^(m×r) and A ∈ ℝ^(r×n) are not required to be low-rank. This is because they remain static and we do not need to store their values."

This reparameterization is the paper's core conceptual contribution: it shows that learning what to scale is more parameter-efficient than learning what directions to update, at least for the adaptation tasks studied. The scaling vectors d can be understood as learning which rows of the frozen random projection are useful (and with what sign and magnitude), while b scales the final output of the adapted subnetwork.

The Inference-Time Advantage: Merged Weights

A crucial practical requirement that the paper preserves from LoRA is zero inference-time overhead. Like LoRA, VeRA's adaptation can be fully merged into the pretrained weights after training. The forward pass in VeRA is:

h = W₀x + Λ_b B Λ_d A x

where Λ_b and Λ_d are diagonal matrices formed from the learned scaling vectors. After training, the term Λ_b B Λ_d A is a fixed matrix (of shape m × n, the same as W₀) that can be pre-computed and added to W₀. The inference forward pass becomes simply h = (W₀ + ΔW)x — identical in cost to the unadapted model.

This distinguishes VeRA from adapter-based methods, which introduce additional computation in the forward pass that cannot be merged away. The paper explicitly highlights this in Section 3:

"Similarly to LoRA, trained scaling vectors along with low-rank matrices can be merged into original weights, eliminating additional inference latency."

The Memory Advantage: Seed-Based Regeneration

The paper's most significant practical innovation is the observation that because the frozen matrices A and B are randomly initialized and not trained, they can be regenerated from a random number generator (RNG) seed rather than stored. Section 3.2 explains:

"Because the random frozen matrices can be regenerated from a random number generator (RNG) seed, these do not need to be stored in memory. This substantially reduces the memory requirement, which is now limited to the bytes needed for the trained b and d vectors and a single RNG seed."

This is the linchpin of VeRA's storage advantage. In LoRA, every adapted model requires storing both A and B matrices (r × (m + n) values). In VeRA, only the scaling vectors d (r values) and b (m values) need to be stored per layer, plus a single seed to regenerate the shared frozen matrices. The parameter count comparison in Table 1 makes this stark: for GPT-3 at rank 16, LoRA requires 75.5M stored parameters (~288 MB) while VeRA requires 2.8M (~10.5 MB) — a 27× reduction. At scale, this difference determines whether per-user model storage is feasible at all.

The paper frames this as directly addressing the motivating scenario from the introduction:

"the main advantage of VeRA is its minimal memory footprint for storing the trained weight adjustments... many versions can reside in the limited memory of a single GPU, thus substantially improving serving efficiency and removing the bottleneck of loading specific models into memory."

What the Paper Explicitly Does Not Address

The paper is careful about its scope. Section 5 acknowledges that "the applicability of the method across different architectures and domains remains an area for future research" — all experiments use Transformer architectures (RoBERTa, GPT-2, Llama, ViT), and the generalization to other architectures (convolutional networks, state-space models, mixture-of-experts) is untested. The paper also notes that "the performance of the method may benefit from additional refinements, such as dynamic parameter budget allocation, or different initialization and regularization techniques" — suggesting that the current results may be a lower bound on what the approach can achieve with further optimization.

The paper additionally does not claim that VeRA is universally superior to LoRA. The results show competitive or slightly better performance on the evaluated benchmarks, but the primary claim is about parameter efficiency at matched performance, not about absolute performance gains. This is a realistic framing: VeRA is positioned as a method that achieves comparable quality with dramatically lower storage requirements, not as a method that fundamentally improves model capabilities.

3. Technical Approach

3.1 Reader Orientation

This is primarily a method paper proposing a new parameterization for parameter-efficient finetuning. VeRA is a finetuning system that adapts large pretrained models to new tasks by learning only small scaling vectors that modulate frozen, randomly initialized, and layer-shared low-rank matrices, rather than learning the low-rank matrices themselves. The problem it solves is the storage bottleneck in deployments requiring many adapted model versions — by making the adaptation so tiny that it can be stored in kilobytes rather than megabytes per model, while the bulky frozen matrices are regenerated on-the-fly from a single random seed shared across all adaptations.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that interact during finetuning and then collapse into a single weight matrix at inference:

Pretrained weight matrix (W₀) — the frozen, original weights of the large language or vision model. These are never updated during finetuning and serve as the backbone that provides most of the model's capabilities.
Frozen random matrices (A and B) — a single pair of randomly initialized matrices shared across all adapted layers. Matrix A has shape r × n (projects input down to the rank dimension) and matrix B has shape m × r (projects back up to output dimension). These are generated once from a known random seed and never trained. Because they are shared, the same A and B serve every transformer layer's query and value projections.
Trainable scaling vectors (d and b) — the only parameters that receive gradient updates. Vector d ∈ ℝ^(1×r) modulates the rows of A (or equivalently, scales each column of A's contribution) before the input is projected through B. Vector b ∈ ℝ^(m×1) scales each row of the final constructed adaptation matrix. Together, these two vectors control which directions in the random basis are amplified, suppressed, or inverted (via sign changes), enabling layer-specific adaptation despite the shared frozen matrices.
The merged adaptation ΔW — after training, the entire adaptation mechanism is collapsed into a single matrix Λ_b B Λ_d A, which has shape m × n, identical to the pretrained weight. This is added to W₀ to produce the final adapted weight. At inference, the model simply uses W = W₀ + ΔW, with zero additional computation compared to the unadapted model.

Information flow during finetuning: An input activation x arrives at a layer → it is multiplied by the frozen pretrained weight W₀ (producing the base output) → simultaneously, x is multiplied by the frozen random matrix A → the result is element-wise scaled by the trainable vector d (via diagonal matrix Λ_d) → this scaled result is multiplied by the frozen random matrix B → the output is element-wise scaled by the trainable vector b (via diagonal matrix Λ_b) → the two paths are summed: h = W₀x + Λ_b B Λ_d A x. Only d and b receive gradient updates; W₀, A, and B are frozen throughout.

Information flow during deployment: The product Λ_b B Λ_d A is pre-computed once per adapted model and stored as a dense matrix (or the tiny d and b vectors are stored and the product is regenerated on loading). The adapted model's forward pass becomes indistinguishable from the original model's — just a single matrix multiply per adapted layer.

3.3 Roadmap for the Deep Dive

First, the formal mathematical formulation of VeRA (Equation 2), which defines exactly how the frozen matrices and trainable scaling vectors compose to produce the weight update, and how this differs from LoRA's formulation.
Second, the parameter count analysis, which explains why VeRA's parameter count scales as L_tuned × (d_model + r) versus LoRA's 2 × L_tuned × d_model × r, and why this matters for storage at scale.
Third, the initialization strategies for both the frozen matrices and the scaling vectors, which are critical to training stability — including why Kaiming initialization is chosen, why b is initialized to zeros, and how the initial value of d serves as a crucial hyperparameter.
Fourth, the mechanism by which the adaptation is merged into the pretrained weights for zero-cost inference, and the seed-based regeneration strategy that eliminates the need to store the frozen matrices.
Fifth, a walkthrough of the training procedure — what gets optimized, how gradients flow, and what hyperparameters govern the process across the different experimental settings (GLUE, E2E, instruction-tuning, image classification).

3.4 Detailed, Sentence-Based Technical Breakdown

This is a method paper whose core idea is that the weight update during finetuning can be reparameterized from "learn the low-rank basis matrices" (LoRA) to "learn scaling coefficients on a fixed random basis that is shared across all layers," reducing the number of stored parameters per adapted model from O(r × d_model) to O(r + d_model) without sacrificing performance.

The Forward Pass: How VeRA Constructs the Weight Update

The paper's central equation (Equation 2) defines the forward pass through an adapted layer:

$h = W_0 x + \Delta W x = W_0 x + \Lambda_b B \Lambda_d A x$

where $W_0 \in \mathbb{R}^{m \times n}$ is the frozen pretrained weight matrix for the layer (e.g., a query or value projection in a self-attention module), $x$ is the input activation vector, $A \in \mathbb{R}^{r \times n}$ is a frozen random matrix that projects from the input dimension to the rank dimension, $B \in \mathbb{R}^{m \times r}$ is a frozen random matrix that projects from the rank dimension to the output dimension, $\Lambda_d$ is a diagonal matrix formed from the trainable vector $d \in \mathbb{R}^{1 \times r}$ (so $\Lambda_d$ has shape $r \times r$ with $d_i$ on the diagonal), and $\Lambda_b$ is a diagonal matrix formed from the trainable vector $b \in \mathbb{R}^{m \times 1}$ (so $\Lambda_b$ has shape $m \times m$ with $b_j$ on the diagonal).

What it computes: The output $h$ is the sum of two terms. The first term $W_0 x$ is the standard forward pass through the frozen pretrained weights — this provides the model's base behavior. The second term $\Lambda_b B \Lambda_d A x$ is the learned adaptation, constructed by a four-step pipeline: (1) the input $x$ is projected through the frozen random matrix $A$ , producing an $r$ -dimensional intermediate vector; (2) each element of this intermediate vector is multiplied by the corresponding element of the trainable scaling vector $d$ (via the diagonal matrix $\Lambda_d$ ), which can amplify, attenuate, or flip the sign of each dimension; (3) this scaled intermediate is projected through the frozen random matrix $B$ , producing an $m$ -dimensional output; (4) each element of this output is multiplied by the corresponding element of the trainable scaling vector $b$ (via the diagonal matrix $\Lambda_b$ ), providing a second layer of per-dimension scaling. The final adaptation $\Delta W x$ is added to the base output, so the model can either reinforce or counteract the pretrained behavior on a per-example basis.

Why this form: This parameterization separates what directions are available (determined by the frozen random matrices $A$ and $B$ ) from how much each direction is used (determined by the learned scaling vectors $d$ and $b$ ). In LoRA, the equivalent equation is $h = W_0 x + B A x$ where both $A$ and $B$ are learned — meaning the method must discover useful directions from scratch, requiring $r \times (m + n)$ trainable parameters. In VeRA, the random initialization provides a basis that spans a subspace of the full update space, and the learning task reduces to selecting which basis vectors to use and at what magnitude. This is a dramatically lower-dimensional optimization problem: the trainable parameters are only the $r$ entries of $d$ (one per basis direction) plus the $m$ entries of $b$ (one per output dimension). The scaling vector $d$ modulates which rows of $A$ contribute to the intermediate representation — if $d_i = 0$ , the $i$ -th row of $A$ is effectively disabled. Similarly, $b$ modulates which rows of the final constructed matrix $B \Lambda_d A$ are active. Together, these two vectors provide a mechanism for layer-specific adaptation even though $A$ and $B$ are shared across all adapted layers: different layers can learn different $d$ and $b$ values, selectively amplifying different subsets of the shared random basis. The alternative — having each layer learn its own $A$ and $B$ — would require storing $L_{tuned} \times r \times (m + n)$ parameters, defeating the purpose of the method.

A crucial property of this form is that $A$ and $B$ are not required to be low-rank matrices in the traditional sense. The paper states in Section 3.1:

"Note that in this setup, B ∈ ℝ^(m×r) and A ∈ ℝ^(r×n) are not required to be low-rank. This is because they remain static and we do not need to store their values."

This is a subtle but important point. In LoRA, $r$ must be small because both $A$ and $B$ must be stored (cost scales as $r \times (m + n)$ ). In VeRA, $r$ can be large (the paper uses $r = 1024$ for RoBERTa-base) because $A$ and $B$ are regenerated from a seed at zero storage cost. The only cost of increasing $r$ is a linear increase in the size of $d$ (from $r$ to $r + \Delta r$ ). This means VeRA can use a much richer random basis than LoRA could afford to store, giving the scaling vectors more directions to choose from. The learning task is "select and scale from a large, fixed buffet of random directions" rather than "learn a small set of directions from scratch."

Parameter Count: Why VeRA Scales Better

The paper provides a precise accounting of trainable parameters in Section 3.2. Let $L_{tuned}$ be the number of transformer layers to which the adaptation is applied (across query and value projections, so typically $2 \times L_{layers}$ ), let $d_{model}$ be the hidden dimension of these projections (e.g., 768 for RoBERTa-base, 1024 for RoBERTa-large), and let $r$ be the rank hyperparameter. Then:

$\text{|Θ|}_{VeRA} = L_{tuned} \times (d_{model} + r)$

$\text{|Θ|}_{LoRA} = 2 \times L_{tuned} \times d_{model} \times r$

where $|\Theta|_{VeRA}$ is the number of trainable parameters for VeRA and $|\Theta|_{LoRA}$ is the number for LoRA.

What these compute: For VeRA, each adapted layer has one scaling vector $d$ of length $r$ (one coefficient per rank dimension) and one scaling vector $b$ of length $d_{model}$ (one coefficient per output dimension), for a total of $d_{model} + r$ trainable scalars per adapted projection. Multiplied by the number of adapted projections $L_{tuned}$ , this gives the total. For LoRA, each adapted layer has two learned matrices: $A$ with $r \times d_{model}$ entries and $B$ with $d_{model} \times r$ entries (the paper uses the convention that $A \in \mathbb{R}^{r \times d_{model}}$ and $B \in \mathbb{R}^{d_{model} \times r}$ , so each has $d_{model} \times r$ parameters), for a total of $2 \times d_{model} \times r$ per adapted projection.

Why this comparison matters: The scaling behavior as $r$ increases is qualitatively different. In VeRA, increasing $r$ by 1 adds only $L_{tuned}$ parameters (one new entry in each layer's $d$ vector). In LoRA, increasing $r$ by 1 adds $2 \times L_{tuned} \times d_{model}$ parameters (new rows and columns in every layer's $A$ and $B$ matrices). The ratio of LoRA parameters to VeRA parameters is approximately $2 \times d_{model} \times r / (d_{model} + r)$ , which for typical values (e.g., $d_{model} = 768$ , $r = 16$ ) gives roughly $2 \times 768 \times 16 / 784 \approx 31$ — a 31× reduction. This ratio grows with $d_{model}$ , making VeRA increasingly advantageous for wider models. For GPT-3 with $d_{model} = 12{,}288$ and $r = 16$ , LoRA requires $2 \times 96 \times 12{,}288 \times 16 = 37.7$ million parameters per adapted module type, while VeRA requires $96 \times (12{,}288 + 16) = 1.18$ million parameters — a roughly 32× reduction.

The paper makes the storage advantage concrete in Table 1. For GPT-3 with rank 16 applied to query and key layers, LoRA requires storing 75.5M parameters (288 MB in single-precision), while VeRA requires storing only 2.8M parameters (10.5 MB). At rank 256, the gap widens: LoRA requires 1.21 billion parameters (4.6 GB) while VeRA requires 8.7M (33 MB). The key insight is that VeRA's stored parameter count grows only with $L_{tuned} \times r$ (the size of all $d$ vectors), not with $L_{tuned} \times d_{model} \times r$ (the size of the frozen matrices, which are regenerated rather than stored).

The seed-based regeneration mechanism. Section 3.2 explains the storage advantage:

"Because the random frozen matrices can be regenerated from a random number generator (RNG) seed, these do not need to be stored in memory. This substantially reduces the memory requirement, which is now limited to the bytes needed for the trained b and d vectors and a single RNG seed."

In practice, this means that when loading a VeRA-adapted model, the system: (1) loads the base pretrained weights $W_0$ once (shared across all adaptations), (2) uses the stored RNG seed to regenerate the frozen matrices $A$ and $B$ (identical for all adapted layers), (3) loads the tiny $d$ and $b$ vectors specific to this adaptation, (4) computes $\Lambda_b B \Lambda_d A$ for each layer and adds it to $W_0$ . The regeneration step costs some FLOPs at load time but eliminates the dominant storage cost (the $r \times (m + n)$ frozen matrix entries). For a deployment with one million adapted models, the storage saving is $10^6 \times$ (what LoRA would store for $A$ and $B$ ), which the introduction estimates at 275 TB for GPT-3 rank-16 LoRA.

Initialization Strategies: Ensuring Trainability from the Start

The paper specifies three initialization choices that are critical to making the training dynamics work correctly. These are described in Section 3.3.

Frozen matrices $A$ and $B$ : Kaiming initialization. The paper states:

"In our method, we employ Kaiming initialization (He et al., 2015) for the frozen low-rank matrices A and B. By scaling the values based on matrix dimensions, it ensures that a matrix product of A and B maintains a consistent variance for all ranks, eliminating the need to finetune the learning rate for each rank."

Kaiming initialization (also known as He initialization) draws each entry of $A$ and $B$ from a zero-mean normal distribution with variance scaled by the fan-in of the layer. For a matrix with input dimension $d_{in}$ , the variance is $2 / d_{in}$ (for ReLU activations) or $1 / d_{in}$ (for linear activations). In VeRA's case, $A$ has shape $r \times n$ so each entry is drawn from $\mathcal{N}(0, \sigma^2)$ where $\sigma^2$ depends on $n$ , and $B$ has shape $m \times r$ so $\sigma^2$ depends on $r$ .

Why this matters: The product $BA$ (or $\Lambda_b B \Lambda_d A$ after scaling) will have entries whose variance is approximately independent of the choice of $r$ . This is critical because $r$ varies across experiments (from 256 to 1024 in the paper), and the paper does not want to re-tune the learning rate for each rank. If the initialization variance grew with $r$ , the effective step size would change with the rank, requiring a new learning rate sweep for each configuration. Kaiming initialization normalizes this away.

The paper also explores alternatives in the ablation study (Table 6b). Kaiming uniform initialization slightly outperforms Kaiming normal on MRPC (90.5 vs. 90.0) and RTE (85.8 vs. 82.6), while a naive uniform initialization in $[0, 0.1]$ performs dramatically worse (68.9 MRPC, 53.1 RTE). This confirms that proper variance scaling is essential — the network cannot adapt effectively if the random basis vectors have the wrong magnitude.

Scaling vector $b$ : zero initialization. The paper initializes $b$ to zeros, matching the standard practice from LoRA where the $B$ matrix is initialized to zero so that $\Delta W = 0$ at the start of training. Section 3.3 states:

"The scaling vector b is initialized to zeros, which aligns with the initialization of matrix B in LoRA and ensures that the weight matrix is unaffected during the first forward pass."

Why this matters: This is a standard technique in residual adaptation methods. If $\Delta W$ started with non-zero values, the model's behavior would be altered before any training occurs, potentially moving the pretrained features away from their optimal point and causing training instability in the early steps. By starting with $b = 0$ , the adaptation term $\Lambda_b B \Lambda_d A x$ is identically zero regardless of the values of $A$ , $B$ , and $d$ . The model begins training from exactly the pretrained state, and $b$ gradually moves away from zero as the optimizer discovers useful directions.

Scaling vector $d$ : a non-zero constant. The paper states:

"The scaling vector d is initialized with a single non-zero value across all its elements, thereby introducing a new hyperparameter that may be tuned for better performance."

The paper explores three values for $d_{init}$ : $10^{-1}$ , $10^{-7}$ , and $1.0$ . The ablation study (Table 6c) shows that $10^{-1}$ and $10^{-7}$ both perform well, while $1.0$ degrades performance substantially (70.3 on MRPC vs. 90.5, 60.3 on RTE vs. 85.8).

Why this matters: The value of $d_{init}$ controls the effective magnitude of the adaptation signal in the early steps of training. When $d$ is initialized to a very small value ( $10^{-7}$ ), the adaptation term $\Lambda_b B \Lambda_d A x$ is near-zero even after $b$ begins to move away from zero — this provides a "slow start" that prevents the model from making large, potentially destructive updates before it has identified useful directions. When $d$ is initialized to $1.0$ , the early updates can be large and may push the model into poor regions of parameter space from which recovery is difficult. The paper hypothesizes:

"values $10^{-1}$ and $10^{-7}$ outperformed $1.0$ , potentially offering more flexibility in the optimization process through early sign changes in selected rows of the frozen matrices."

The concept of "early sign changes" is important: if $d_i$ starts at a small positive value, the optimizer can easily drive it negative (flipping the sign of the $i$ -th basis direction) with only a few gradient steps. If $d_i$ starts at $1.0$ , it takes many more steps to flip the sign, effectively making sign changes (which correspond to inverting the contribution of a basis vector) more difficult to learn.

Why a separate hyperparameter $d_{init}$ rather than incorporating it into the learning rate: The learning rate controls the step size for all parameters uniformly. The $d_{init}$ hyperparameter specifically controls the initial scale of the $d$ vector relative to the $b$ vector and the frozen matrices, allowing the practitioner to tune the "aggressiveness" of the early adaptation independently from the overall learning rate. This is a design choice that trades a small amount of additional tuning burden for substantially more control over training dynamics. The paper uses $d_{init} = 0.1$ as the default throughout the main experiments (Table 8).

Merging into Pretrained Weights for Zero-Cost Inference

The paper emphasizes that VeRA, like LoRA, incurs no additional inference latency because the entire adaptation can be pre-computed and merged into the frozen weights. Section 3 states:

"Similarly to LoRA, trained scaling vectors along with low-rank matrices can be merged into original weights, eliminating additional inference latency."

The merging procedure is straightforward. After training is complete, for each adapted layer, the system computes:

$\Delta W = \Lambda_b B \Lambda_d A$

where $\Delta W$ is the effective weight update matrix of shape $m \times n$ (same as $W_0$ ). This is a dense matrix — it costs $m \times n$ FLOPs to compute once, but this is a one-time cost at model loading time, not per forward pass. The adapted weight becomes:

$W_{adapted} = W_0 + \Delta W$

At inference, the forward pass is simply $h = W_{adapted} x$ — a single matrix-vector multiply identical in cost to the original unadapted model. There is no additional computation, no extra layers, and no residual connections beyond the standard transformer architecture.

Why this matters for deployment: This property distinguishes VeRA from adapter-based methods (Houlsby et al., 2019; Lin et al., 2020; Pfeiffer et al., 2021), which insert additional trainable layers into the forward pass. Adapter layers cannot be merged into the pretrained weights because they include non-linearities (e.g., ReLU activations between the down-projection and up-projection). They must be executed sequentially with the pretrained layers, adding latency to every forward pass. VeRA's adaptation is purely linear (a product of matrices and diagonal scaling matrices), so the entire adaptation collapses into a single matrix that can be fused with the pretrained weights.

The merging operation is also what enables the seed-based storage strategy. If VeRA had to store the full $\Delta W$ matrix for each adapted model, the storage cost would be $L_{tuned} \times m \times n$ , which is actually larger than LoRA's storage cost (since LoRA stores only the low-rank factors, not the full-rank product). The key insight is that the merging can be done at load time from the tiny stored vectors and the regenerated frozen matrices, so the storage is proportional to the stored vectors (kilobytes to megabytes) rather than the merged matrix (potentially gigabytes).

Training Procedure and Hyperparameters

The paper applies VeRA across four distinct experimental settings, each with its own hyperparameter configuration. The training procedure is standard supervised finetuning — the model receives labeled examples, computes a task-specific loss (cross-entropy for classification, language modeling loss for generation), and gradients flow only to the scaling vectors $d$ and $b$ (and the task-specific classification head, which is trained fully). The pretrained weights $W_0$ and the frozen random matrices $A$ and $B$ receive no gradient updates.

Which layers are adapted. The paper applies VeRA to the query and value projection matrices in each self-attention module, following the standard practice from LoRA. The paper does not adapt the key projections, the output projection, or the feed-forward network layers. The classification head (a linear layer mapping from the model's hidden state to the output classes) is always trained fully, and its parameters are excluded from the reported trainable parameter counts.

GLUE benchmark (Section 4.1). For RoBERTa-base, the paper uses $r = 1024$ for VeRA. For RoBERTa-large, $r = 256$ . The rank choice is essentially a hyperparameter that controls the richness of the random basis — higher $r$ provides more basis directions for the scaling vectors to select from, at the cost of a linear increase in the size of the $d$ vectors. The paper selects these ranks empirically to achieve competitive performance. All hyperparameters are listed in Table 8:

Optimizer: AdamW
Warmup ratio: 0.06
Learning rate schedule: Linear
Initialization of shared matrices: Kaiming Uniform
Initial value of $d$ : 0.1
Batch size: 64 for RoBERTa-base, 32 for RoBERTa-large
Maximum sequence length: 512 for RoBERTa-base, 128 for RoBERTa-large
Separate learning rates for the classification head and the VeRA-adapted layers, tuned per task (Table 8). For example, on SST-2 with RoBERTa-base: head learning rate $4 \times 10^{-3}$ , VeRA learning rate $4 \times 10^{-3}$ . On RTE with RoBERTa-large: head learning rate $2 \times 10^{-3}$ , VeRA learning rate $2 \times 10^{-2}$ .

The paper notes that unlike LoRA, which uses an additional hyperparameter $\alpha$ to scale the adaptation term (where the effective update is $\frac{\alpha}{r} B A$ ), VeRA introduces separate learning rates for the classification head and adapted layers instead. The justification is not explicitly stated, but the separate learning rates serve a similar purpose: they allow the practitioner to control the relative speed of adaptation versus head training. This is important because the classification head starts from random initialization while the adaptation starts from the pretrained state (via $b = 0$ ), so they may benefit from different step sizes.

The paper acknowledges that due to academic compute constraints, it was not possible to run full grid searches on all hyperparameters:

"Note that due to our academic compute we were not able to run full grid searches on any hyperparameters. We only evaluated different learning rates and number of epochs and even relied on existing configurations of LoRA (Optimizer, Warmup ratio, LR schedule)."

E2E benchmark (Section 4.2). For GPT-2 Medium and Large on the E2E natural language generation benchmark, the paper uses $r = 1024$ for both model sizes. The hyperparameters (Table 10) are:

Optimizer: AdamW
Learning rate schedule: Linear
Weight decay: 0.01
Batch size: 8
Epochs: 5
Warmup steps: 500
Label smoothing: 0.1
Learning rate: $1 \times 10^{-1}$ for GPT-2 Medium, $2 \times 10^{-2}$ for GPT-2 Large

The paper follows the same experimental setup as LoRA for E2E, changing only the rank and learning rate. Unlike GLUE, the E2E task is a generation task rather than a classification task, so there is no separate classification head — the model's output logits over the vocabulary are used directly to compute the language modeling loss.

Instruction-tuning (Section 4.3). For Llama 7B and 13B models, the paper applies VeRA to "all linear layers except the top one, similarly to Dettmers et al. (2023)." This is a broader application than GLUE (which only adapted query and value layers), covering all linear projections in the self-attention and feed-forward network modules. The hyperparameters (Table 9) are:

Rank $r$ : 1024 (for both 7B and 13B)
Optimizer: AdamW
Warmup ratio: 0.1
Batch size: 4
Accumulation steps: 4 (effective batch size of 16)
Epochs: 1
Learning rate schedule: Cosine
Learning rate: $4 \times 10^{-3}$ (compared to $4 \times 10^{-4}$ for LoRA in the same setting)

The paper additionally leverages quantization techniques from Dettmers et al. (2023) to train the model on a single GPU. This means the pretrained weights are stored in 4-bit precision, further reducing memory requirements during training, while the scaling vectors are trained in full precision or 16-bit.

Image classification (Section 4.4). For Vision Transformers (ViT-Base and ViT-Large) on four image classification datasets, the paper applies VeRA to the query and value layers, matching the application pattern from GLUE. The hyperparameters (Table 11) include:

VeRA rank: 256
LoRA rank (for comparison): 8
Optimizer: AdamW
Learning rate schedule: Linear
Weight decay: 0.01
Separate learning rates for the classification head and the adapted layers, tuned per dataset. For example, on CIFAR100 with ViT-Base: head LR $4 \times 10^{-3}$ , VeRA LR $2 \times 10^{-2}$ .

The paper trains on a subset of 10 samples per class and evaluates on the full test set, a few-shot adaptation setting that tests whether the method can quickly adapt to new visual domains with minimal data.

The Scaling Vector Mechanism in Detail

The paper's ablation study (Section 4.6, Table 6a) provides insight into why both $d$ and $b$ are necessary and what each vector contributes to the adaptation. Two ablation setups are tested: "only $d$ " (where $b$ is removed, so the forward pass is $h = W_0 x + B \Lambda_d A x$ ) and "only $b$ " (where $d$ is removed, so the forward pass is $h = W_0 x + \Lambda_b B A x$ ).

Results on RoBERTa-large:

Full VeRA: MRPC 90.5 ± 0.7, RTE 85.8 ± 0.7
Only $d$ : MRPC 89.7 ± 0.0, RTE 67.0 ± 13.9
Only $b$ : MRPC 81.6 ± 10.1, RTE 64.3 ± 11.5

The "only $d$ " configuration retains most of the performance on MRPC but degrades on RTE, while "only $b$ " degrades significantly on both tasks. The paper interprets this asymmetry:

"This disparity in performance underscores the higher expressiveness of the d scaling vector over the b vector. Specifically, d modulates the rows of both low-rank matrices, thereby influencing a broader aspect of the final constructed matrix. In contrast, b only scales the rows of the final matrix resulting from the product of the low-rank matrices."

What this means operationally: In the full VeRA forward pass $\Lambda_b B \Lambda_d A x$ , the vector $d$ sits between $A$ and $B$ in the computation graph. It scales the intermediate representation after the first projection but before the second projection. This means $d$ controls which directions in the random basis $A$ are emphasized when they are combined by $B$ . Changing $d_i$ changes how much the $i$ -th row of $A$ contributes to all output dimensions (since $B$ mixes these contributions). In contrast, $b$ sits at the very end, scaling each output dimension independently after the entire $B \Lambda_d A$ computation is complete. The $d$ vector thus has a more global influence — it can disable an entire basis direction across all outputs — while $b$ can only attenuate specific output dimensions after the basis directions have already been combined.

The high variance in the "only $d$ " and "only $b$ " results (standard deviations of 10–14 on RTE) suggests that these ablated configurations are unstable — they sometimes train well and sometimes fail completely, depending on the random seed. The full VeRA with both vectors provides more robust training dynamics.

Layer-Wise Adaptation Magnitudes

The paper visualizes how the learned $d$ vectors differ across layers after finetuning on the RTE task (Figure 3). Because the frozen matrices $A$ and $B$ are identical across all layers, the $d$ vector's L2 norm ( $||d||_2$ ) is directly comparable across layers — a larger norm means that layer is applying a larger-magnitude adaptation to the shared random basis.

The figure shows two patterns:

Query matrices receive larger adaptation than value matrices — the $||d||_2$ values for query projections are consistently higher across all layers, suggesting that adapting how the model computes attention patterns (query) is more impactful than adapting how it aggregates values (value).
Later layers receive larger adaptation than earlier layers — the $||d||_2$ values increase roughly monotonically from layer 0 to layer 23, with the final layers showing 2–3× larger magnitude than the initial layers.

This second pattern aligns with findings from prior work on efficient adaptation (Zhang et al., 2023b; Liu et al., 2021), which observed that later transformer layers benefit more from finetuning than earlier layers. The paper's contribution is showing that VeRA's scaling vectors naturally capture this layer-wise variation despite the frozen matrices being identical across layers — the per-layer $d$ and $b$ vectors learn to apply different amounts of adaptation to the same random basis.

The paper conducts an ablation to determine whether sharing the frozen matrices $A$ and $B$ across all adapted layers harms performance compared to having unique per-layer random matrices (Table 7). Across four GLUE tasks (MRPC, RTE, CoLA, STS-B), the results are:

Task	Shared Matrices	Unique Matrices
MRPC	90.0 ± 0.9	90.7 ± 0.3
RTE	84.6 ± 1.5	84.6 ± 0.8
CoLA	67.7 ± 0.8	68.3 ± 1.8
STS-B	91.5 ± 0.6	91.5 ± 0.2

What this shows: The performance difference between shared and unique frozen matrices is negligible — within one standard deviation for all tasks. This validates the paper's central design choice: sharing the frozen matrices across layers does not meaningfully constrain the model's ability to adapt, because the per-layer scaling vectors $d$ and $b$ provide sufficient layer-specific modulation. The frozen matrices provide a universal random basis, and different layers learn to select and scale different subsets of that basis.

Why this matters for storage: If unique matrices were necessary, the storage advantage would partially evaporate — the frozen matrices could still be regenerated from seeds, but each layer would need its own seed and its own matrices, and the total regeneration cost at load time would scale with the number of layers. By sharing matrices, the system regenerates one pair of matrices once, then applies them to all layers with per-layer scaling. The stored state is just the $d$ and $b$ vectors per layer (plus one global seed).

Relationship to Prior Work: The Connection to Random Projections

The paper frames VeRA within the literature on random projections and frozen models (Section 2). While the method is novel, the underlying principle — that a fixed random basis can support learning when properly scaled — has precedents:

Aghajanyan et al. (2021) showed that finetuning can be performed in a randomly projected low-dimensional subspace, achieving 90% of full finetuning performance. VeRA inverts this: rather than projecting the parameter space into a low-dimensional random subspace and optimizing there, VeRA projects the adaptation through a high-dimensional random basis and optimizes only the scaling of those basis vectors.

Ramanujan et al. (2020) demonstrated that randomly weighted neural networks contain subnetworks that perform well without any training. VeRA's frozen matrices can be seen as providing a similar "reservoir" of useful random features, and the scaling vectors select which of these features to activate for a given task.

Frankle et al. (2021) showed that training only batch normalization parameters in a randomly weighted network can achieve non-trivial performance. VeRA's scaling vectors $d$ and $b$ serve an analogous role to batch normalization parameters — they modulate the magnitude and sign of random features without changing the features themselves.

Ruiz et al. (2023) used frozen random matrices inside LoRA for text-to-image model personalization. VeRA extends this idea by sharing the frozen matrices across layers and applying it to language understanding, generation, and instruction-following tasks, demonstrating broader applicability.

VeRA distinguishes itself from these precedents by (1) sharing the frozen matrices across all layers rather than having per-layer random bases, (2) using two scaling vectors ( $d$ and $b$ ) operating at different points in the computation, and (3) applying the approach systematically across multiple domains and model scales with competitive results against the state-of-the-art.

Training Memory and Time Overhead

The paper provides a practical comparison of training efficiency between LoRA and VeRA when finetuning Llama 7B at the same rank ( $r = 64$ ) on the instruction-tuning dataset (Appendix C, Table 12):

LoRA training time: 568 minutes
VeRA training time: 578 minutes (1.8% increase)
LoRA GPU memory: 23.42 GB
VeRA GPU memory: 21.69 GB (7.4% reduction)

The slight increase in training time is attributed to the additional vector multiplications in the forward pass ( $\Lambda_d$ and $\Lambda_b$ ), which add operations not present in LoRA. However, the 1.8% overhead is modest and unlikely to be a practical concern. The 7.4% reduction in GPU memory is because VeRA does not need to store optimizer states (momentum and variance terms in AdamW) for the frozen matrices $A$ and $B$ — only the tiny scaling vectors require optimizer states. For LoRA at rank 64 on a 7B model, the optimizer states for $A$ and $B$ are substantial (each parameter has two AdamW state variables), so eliminating them provides a measurable memory saving.

This tradeoff — marginally more computation for meaningfully less memory — aligns with VeRA's design philosophy: prioritize storage and memory efficiency over computational efficiency, since the deployment bottleneck is storage (many adapted models) and training memory (fitting on a single GPU), not per-step FLOPs.

4. Key Insights and Innovations

Innovation 1: Reparameterizing Adaptation as "Scale Selection" Rather Than "Basis Discovery"

The dominant conceptual framework in parameter-efficient finetuning, established by LoRA (Hu et al., 2022), treats the adaptation problem as learning a low-rank basis: the method must discover which directions in the weight update space are useful for the target task, and the learned matrices A and B represent that discovered basis. This framing is natural — it mirrors how we think about dimensionality reduction and matrix factorization — but it commits the method to storing the learned basis vectors for every adapted model, since each task or user might need different directions.

VeRA makes a genuinely different conceptual move: decouple the basis from the adaptation entirely. The basis (frozen random matrices A and B) is chosen once, arbitrarily, and shared across all layers and all adaptations. The learning task reduces to selecting which basis vectors to use and at what scale, implemented through the trainable diagonal scaling vectors d and b. This is not an optimization trick or a compression of LoRA — it is a fundamentally different decomposition of the adaptation problem. In LoRA, the question is "what low-dimensional subspace should the weight update live in?" In VeRA, the question is "given a high-dimensional random subspace, how should I weight its components?"

The intellectual shift is subtle but profound. LoRA's formulation implies that different tasks genuinely need different low-rank subspaces: the directions that help with sentiment analysis differ from those that help with question answering, and the model must learn those directions separately. VeRA's formulation implies that a sufficiently rich random basis spans useful directions for many tasks, and task-specific adaptation is primarily about emphasizing vs. suppressing directions within that shared basis. If this holds — and the paper's results suggest it does across language understanding, generation, instruction following, and vision — it means that most of what we thought of as "task-specific adaptation" is actually task-specific scaling of a universal adaptation basis.

This reframing has two immediate consequences that the paper exploits. First, the basis can be discarded and regenerated because it is not task-specific — only the scaling coefficients are. Second, the rank r can be made very large (r = 1024 is used for RoBERTa-base and instruction-tuning) without storage penalty, because the basis doesn't need to be stored. LoRA's rank is constrained by storage cost; VeRA's rank is constrained only by the acceptable size of the scaling vectors, which grows linearly with r rather than multiplicatively. This means VeRA can operate in a regime — high-rank random basis, tiny learned modulation — that is conceptually inaccessible to LoRA.

The paper provides empirical evidence for this reframing through the cosine similarity analysis in Appendix D (Figure 5). When comparing the effective weight updates learned by LoRA and VeRA at the same rank (r = 64), the cosine similarity is non-trivial (on the order of 10⁻³, two orders of magnitude above the random baseline), and it increases in later layers — exactly where the magnitude analysis (Figure 3) shows the largest adaptation. This suggests VeRA is not doing something entirely different from LoRA; rather, it is approximating similar weight updates through a different parameterization, one that separates basis from scaling. The fact that this approximation works with 100× fewer stored parameters (Table 4) is evidence that the basis-discovery component of LoRA carries substantial redundancy across tasks.

The comparison to prior work sharpens the significance. Aghajanyan et al. (2021) showed that finetuning has low intrinsic dimensionality — you only need to optimize a few hundred parameters if those parameters are projected back into the full weight space through a random matrix. VeRA inverts this logic: rather than compressing the parameter space and projecting back, VeRA keeps a high-dimensional random basis in the weight space and learns to modulate it. The two approaches are dual — Aghajanyan et al. optimize a low-dimensional code projected through a fixed random matrix; VeRA optimizes scaling coefficients on a fixed random matrix embedded in the full weight space. VeRA's version is more practical for deployment because the stored parameters ARE the adaptation (the scaling vectors), not a compressed encoding that must be decompressed at load time.

This is a fundamental conceptual contribution rather than an incremental refinement. It changes how one thinks about what parameter-efficient adaptation needs to store versus what it can regenerate, and it opens a design space — frozen random projections with learned modulation — that extends beyond the specific VeRA formulation (different initialization schemes, different modulation structures, dynamic basis selection) and invites further exploration.

Innovation 2: The Observation That Per-Layer Adaptation Can Emerge from a Shared Basis Plus Tiny Per-Layer Vectors

A key architectural choice in VeRA is that the frozen random matrices A and B are shared across all adapted layers. Every query projection and every value projection in every transformer block uses the same A and B. The only layer-specific parameters are the scaling vectors d and b, each of which is a handful of scalars — r entries for d (typically 256–1024) and d_model entries for b (768–12288 depending on the model). This is a radical departure from LoRA, where each adapted layer learns its own independent low-rank matrices, and from adapter-based methods, where each adapter layer has its own independently trained weights.

The intellectual contribution here is the demonstration that layer specialization does not require layer-specific basis directions. The paper shows — through both the performance results (Table 2, 3, 4, 5) and the layer-wise magnitude analysis (Figure 3) — that different layers CAN adapt differently despite sharing the same random basis, and that this differentiation is achieved purely through the learned scaling vectors. Layer 23 of RoBERTa-large learns a d vector with ~3× larger magnitude than layer 0 (Figure 3), meaning it amplifies the random basis directions much more strongly, even though both layers have access to the exact same set of basis directions. Layer 0 might use basis directions 1–200 with small positive coefficients; layer 23 might use directions 400–600 with large positive coefficients and directions 700–800 with negative coefficients. The basis is identical; the selection and scaling differ.

This finding challenges an implicit assumption in prior work: that different layers need different "types" of adaptation (different subspaces) because they compute different features. VeRA suggests that the space of useful adaptations across layers has substantial overlap — a single random basis, if rich enough, can serve all layers — and that layer-specific behavior emerges from how strongly each layer engages with that shared basis, not from which basis it uses. This is analogous to the finding in multi-task learning that hard parameter sharing can match or exceed task-specific architectures when capacity is sufficient, but applied at the intra-model level: layers are the "tasks," and they share the adaptation basis.

The ablation study in Table 7 directly tests this claim by comparing shared vs. unique random matrices. Across four GLUE tasks, the difference is negligible — shared matrices achieve 90.0 on MRPC vs. 90.7 for unique matrices, and identical performance on RTE and STS-B. This is a striking null result: adding per-layer basis diversity (unique random matrices) does not meaningfully improve performance. The per-layer scaling vectors alone provide sufficient expressivity to capture layer-specific adaptation patterns.

The practical consequence is enormous. If layer-specific bases were necessary, VeRA's storage advantage would shrink dramatically because each layer would need its own random seed to regenerate its unique matrices, and the load-time computation would scale with the number of layers. By sharing the matrices, the system regenerates ONE pair of matrices, applies them everywhere, and stores only the per-layer scaling vectors. The storage cost per adapted model becomes L_tuned × (d_model + r) scalars, which for Llama 13B with r = 1024 is 2.4M parameters (~9.6 MB in single precision) versus LoRA's 250.3M parameters (~1 GB).

This is a fundamental empirical finding with significant architectural implications. It suggests that future parameter-efficient methods should invest representational capacity in how they modulate a shared basis (e.g., more complex scaling functions, non-linear modulation, learned interactions between modulation vectors) rather than in expanding the number of independently parameterized bases per layer.

Innovation 3: Identifying the Practical Bottleneck as Storage, Not Training, and Solving It Through Seed-Based Regeneration

Prior work on parameter-efficient finetuning — including LoRA, AdaLoRA, adapters, and BitFit — focused predominantly on reducing the number of parameters that need to be trained, motivated by the computational cost of gradient computation and optimizer state maintenance for large models. The storage requirements of the adapted models, while mentioned, were treated as a secondary concern. This made sense when the primary deployment scenario was a single organization serving a handful of tasks: storing a few LoRA checkpoints of tens or hundreds of megabytes each is manageable.

VeRA reorients the problem statement. The paper opens not with a claim about training efficiency but with a concrete storage calculation: 275 TB to store one million LoRA-adapted GPT-3 models (Section 1). This reframing — from "how do we train cheaply" to "how do we store and serve millions of adapted models" — changes what counts as a solution. A method that reduces training parameters by 10% but still requires megabytes per adaptation doesn't solve the storage problem. A method that reduces stored parameters by 100×, even if training is slightly slower (the 1.8% overhead in Table 12), does.

The conceptual innovation is decoupling the number of parameters that are trained from the number of parameters that are stored. In every prior PEFT method, these two quantities were essentially the same: you store what you train. VeRA introduces an asymmetry. The frozen matrices A and B are part of the training computation — every forward pass involves multiplying by them, and gradients conceptually flow through them (though they're not updated) — but they are NOT part of the stored adaptation. They are regenerated from a random seed at load time. The stored adaptation is only the scaling vectors d and b, which are orders of magnitude smaller than the matrices they modulate.

This asymmetry is the paper's most significant practical contribution. It means that the cost of adding a new user or task — in terms of persistent storage — is dominated by the size of the scaling vectors (tens to hundreds of kilobytes per layer) rather than the size of the adapted weight matrices. At one million users with Llama 13B, LoRA (rank 64) requires roughly 1 GB per user → 1 PB total; VeRA requires roughly 10 MB per user → 10 TB total. The difference between "needs a data center" and "fits on a single server's SSDs" is what VeRA's approach enables.

This innovation is incremental in mechanism (the seed-based regeneration trick is simple) but fundamental in its implications for deployment architecture. It changes the economic calculus of model personalization. If storing an adapted model costs 10 MB rather than 1 GB, it becomes feasible to maintain per-user adaptations for millions of users on modest infrastructure. This in turn enables use cases — like the continuously learning personal assistant from the introduction — that were economically infeasible under LoRA's storage requirements.

The paper provides direct evidence for this through the memory comparison in Table 1 and the parameter count formulas in Section 3.2, but the true significance is not in any single experimental result — it's in how the method redefines the Pareto frontier of the storage-performance tradeoff. Figure 2 shows that when VeRA is given the same parameter budget as LoRA (by increasing r until the stored parameter counts match), VeRA substantially OUTPERFORMS LoRA (by ~4 percentage points on RTE). This means VeRA doesn't just match LoRA with fewer parameters — it achieves better performance at any given storage budget, effectively shifting the entire tradeoff curve.

Innovation 4: Demonstrating That a Single High-Rank Random Basis Suffices Across Tasks, Domains, and Model Scales

The paper's experimental coverage — GLUE natural language understanding, E2E natural language generation, instruction-tuning of Llama models, and image classification with Vision Transformers — is unusually broad for a PEFT method paper. This breadth is not incidental; it serves a specific intellectual purpose: to test whether the "frozen random basis + learned scaling" parameterization is universally applicable or whether it succeeds only in narrow settings with specific architectures.

The results suggest universality within the Transformer family. VeRA matches LoRA on RoBERTa-base and RoBERTa-large for GLUE (Table 2: 85.2 vs. 86.6 average on base, 87.8 vs. 87.8 on large). It matches LoRA on GPT-2 Medium and Large for E2E (Table 3: 70.1 vs. 68.9 BLEU on Medium, 70.3 vs. 70.1 on Large). It closely tracks LoRA on Llama 7B and 13B instruction-tuning (Table 4: 4.77 vs. 5.03 on Llama 7B, 5.22 vs. 5.31 on Llama 13B). It approaches or exceeds LoRA on ViT-Base and ViT-Large image classification (Table 5: 84.8 vs. 85.9 on CIFAR100 for Base, 87.5 vs. 87.0 on CIFAR100 for Large). In every case, the parameter reduction is 3× to 100×.

The intellectual contribution here is establishing that a single random basis, appropriately scaled, can support adaptation across tasks fundamentally different in nature — classification (GLUE), generation (E2E), instruction following (Alpaca → MT-Bench), and visual recognition (CIFAR100, Food101, Flowers102, RESISC45). The basis doesn't need to be task-specific or domain-specific. The frozen matrices A and B are initialized with the same Kaiming uniform distribution regardless of whether they'll be used for sentiment analysis or food classification.

This is a surprising result when viewed against the LoRA paradigm. If different tasks genuinely needed different low-rank subspaces (as LoRA's per-task learned A and B imply), one would expect a shared random basis to work well for some tasks and poorly for others — the random basis might happen to span useful sentiment directions but miss question-answering directions. The fact that it works across such diverse tasks suggests that high-dimensional random subspaces are surprisingly good at containing useful adaptation directions for a wide range of tasks, and the key factor is not the basis itself but the ability to select and scale within it. This aligns with the Johnson-Lindenstrauss lemma intuition that random projections approximately preserve pairwise distances, but extends it to the claim that random projections approximately preserve adaptability — the capacity to shift model behavior in task-relevant ways.

The breadth also serves as an existence proof that VeRA is not exploiting some quirk of the GLUE benchmark, the RoBERTa architecture, or a particular training recipe. The method transfers across encoder-only models (RoBERTa), decoder-only models (GPT-2, Llama), and encoder-decoder Vision Transformers. It works with classification heads (GLUE, image classification), language modeling heads (E2E), and autoregressive generation (instruction-tuning). It works at scales from RoBERTa-base (125M parameters) to Llama 13B (13B parameters). This breadth is strong evidence for the generality of the "scale selection" paradigm and makes it credible that the approach would extend to newer architectures as they emerge.

This contribution is less a single finding and more a comprehensive empirical validation of a conceptual shift. The paper isn't just proposing a method; it's using a broad experimental campaign to argue that the method reflects something fundamental about how adaptation works in pretrained models — that the capacity to adapt is largely about modulation of existing feature directions, not discovery of new ones, and that this modulation can be achieved through scaling of a fixed random basis that spans the space densely enough.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on four distinct benchmarks spanning natural language understanding, natural language generation, instruction following, and image classification. For language understanding, it uses the GLUE benchmark (Wang et al., 2019) with the specific splits from Hu et al. (2022): six tasks — SST-2, MRPC, CoLA, QNLI, RTE, and STS-B — with MNLI and QQP omitted "due to time constraints and budget limitations" (Section 4.1). For language generation, it uses the E2E benchmark (Novikova et al., 2017), a dataset for end-to-end data-to-text generation with approximately 50K training examples. For instruction following, it uses the cleaned version of the Alpaca dataset (Taori et al., 2023), comprising 51K instructions and demonstrations, with evaluation on MT-Bench (Zheng et al., 2023), a set of 80 multi-turn questions scored by GPT-4 on a 1–10 scale. For image classification, it uses four datasets — CIFAR100 (Krizhevsky, 2009), Food101 (Bossard et al., 2014), Flowers102 (Nilsback & Zisserman, 2008), and RESISC45 (Cheng et al., 2017) — training on only 10 samples per class and evaluating on the full test set (or all remaining samples for RESISC45).
Base model(s). For GLUE: RoBERTa-base (125M parameters) and RoBERTa-large (355M parameters) (Liu et al., 2019), both encoder-only Transformer masked language models. For E2E: GPT-2 Medium (355M parameters) and GPT-2 Large (774M parameters) (Radford et al., 2019), decoder-only autoregressive language models. For instruction tuning: Llama 7B and 13B (Touvron et al., 2023a) and Llama2 7B and 13B (Touvron et al., 2023b), decoder-only models. For image classification: ViT-Base (86M parameters) and ViT-Large (307M parameters) (Dosovitskiy et al., 2021), pretrained on ImageNet-21k. These models span encoder-only, decoder-only, and vision Transformer architectures at scales from 125M to 13B parameters, providing broad coverage of the contemporary Transformer landscape. The choice of RoBERTa and GPT-2 specifically follows the standard evaluation protocol established in the original LoRA paper (Hu et al., 2022), enabling direct comparison with reported numbers.
Metrics. GLUE tasks use task-specific metrics: Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for SST-2, MRPC, QNLI, and RTE, with all reported as percentages or correlation coefficients where higher is better. E2E uses five standard NLG metrics: BLEU, NIST, METEOR, ROUGE-L, and CIDEr, with scores from the last training epoch. MT-Bench for instruction tuning uses a single numeric score on a 1–10 scale assigned by GPT-4 acting as a judge, averaged across 80 multi-turn questions, following the protocol of Zheng et al. (2023). Image classification uses standard top-1 accuracy (%). For parameter efficiency comparisons, the paper reports number of trainable parameters and theoretical storage bytes (assuming single-precision, 4 bytes per parameter), explicitly excluding the task-specific classification head from these counts since it must be trained in all methods.
Baselines. For GLUE (Table 2): Full finetuning (all parameters updated), BitFit (Zaken et al., 2022; bias-only finetuning), Adapter tuning variants including AdapterH (Houlsby et al., 2019), AdapterL (Lin et al., 2020), AdapterP (Pfeiffer et al., 2021), and AdapterD (Rücklé et al., 2021), LoRA (Hu et al., 2022) at 0.3M parameters for RoBERTa-base and 0.8M for RoBERTa-large, and LoRA-FA (Zhang et al., 2023a). For E2E (Table 3): Full finetuning, AdapterL and AdapterH at two different parameter budgets, DyLoRA (Valipour et al., 2022), AdaLoRA (Zhang et al., 2023b), and LoRA. For instruction tuning (Table 4): LoRA with rank 64 as proposed by Dettmers et al. (2023), using 159.9M trainable parameters for 7B models and 250.3M for 13B models, plus the unadapted base Llama 13B model as a lower bound. For image classification (Table 5): Head-only (training only the classification head), Full finetuning, and LoRA with rank 8. The multiplicity of baselines on GLUE and E2E is inherited from the LoRA paper's evaluation protocol; the paper did not re-run these baselines but sourced results from prior publications, noting this explicitly in the table captions ("Results of all methods except VeRA are sourced from prior work").
Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of inference-time sampling (this is not a test-time compute scaling paper). For fair comparison of parameter efficiency, the paper measures trainable parameter counts (excluding the classification head) and theoretical storage bytes. Training compute is reported only secondarily: Appendix C (Table 12) provides a direct comparison of training time (568 vs. 578 minutes for LoRA vs. VeRA on Llama 7B) and GPU memory usage (23.42 vs. 21.69 GB) when both methods use the same rank (r = 64), establishing that VeRA's additional vector multiplications incur only a 1.8% training time overhead while providing a 7.4% memory reduction. All other experiments use different ranks for LoRA and VeRA (e.g., LoRA rank 8–64, VeRA rank 256–1024), so direct training cost comparisons at those configurations are not available — the ranks were chosen to optimize each method's performance independently, not to match training cost.
Cross-validation / statistical protocol. For GLUE tasks, the paper performs 5 runs with different random seeds, records the best epoch's outcome for each run, and reports the median of these results, along with standard deviations for VeRA in Table 2. This follows the protocol of Hu et al. (2022). For the scaling experiment in Section 4.5 (Figure 2), the same 5-run median protocol is used. For the ablation study (Section 4.6, Table 6), results are reported as mean and standard deviation across 5 random seeds. For instruction tuning (MT-Bench), image classification, and E2E, the paper reports results from a single training run (no seed-based replication), likely due to the computational cost of these experiments. This is a methodological limitation — single-run results on MT-Bench and E2E are subject to training noise, and the absence of error bars makes it difficult to assess whether the small performance differences (e.g., VeRA at 5.22 vs. LoRA at 5.31 on Llama 13B in Table 4) are statistically meaningful or within run-to-run variance. The paper does not report confidence intervals, statistical significance tests, or the standard deviation for the E2E, MT-Bench, or image classification results.

Main Quantitative Results

GLUE Benchmark: Natural Language Understanding

Table 2 presents the central language understanding comparison. For RoBERTa-base, VeRA achieves 85.2 average across the six GLUE tasks using 43K trainable parameters (0.043M), compared to LoRA's 86.6 average using 300K parameters (0.3M). This is a ~7× reduction in parameters for a 1.4-point performance difference. Per-task, VeRA performs competitively on most tasks: on CoLA it actually outperforms LoRA (65.6 vs. 63.4, a 2.2-point advantage), on MRPC it is comparable (89.5 vs. 89.7), on SST-2 it trails slightly (94.6 vs. 95.1), and on the more challenging reasoning tasks it shows larger gaps — QNLI 91.8 vs. 93.3 (1.5-point gap), RTE 78.7 vs. 86.6 (7.9-point gap). The RTE gap is the largest and suggests that the shared random basis parameterization may struggle when the adaptation requires more task-specific representational changes — RTE is a binary entailment task requiring fine-grained semantic reasoning, arguably the most complex of the GLUE subset.

For RoBERTa-large, the picture shifts: VeRA achieves 87.8 average using 61K parameters (0.061M), which is identical to LoRA's 87.8 average using 800K parameters (0.8M). This is the paper's headline result — a 13× parameter reduction at matched performance. Per-task, VeRA actually outperforms LoRA on MRPC (90.9 vs. 90.2) and RTE (85.9 vs. 85.2, reversing the base-model gap), while trailing slightly on CoLA (68.0 vs. 68.2), QNLI (94.4 vs. 94.8), and STS-B (91.7 vs. 92.3). The closing of the RTE gap from base to large is notable — it suggests that larger models may provide a richer feature space in which the frozen random basis is more effective, possibly because the pretrained features themselves are higher quality and require only subtle modulation rather than major restructuring.

A crucial detail in Table 2: the methods AdptD, AdptP, and AdptH are reported at multiple parameter budgets (e.g., AdptD at 0.3M and 0.9M for RoBERTa-base), while LoRA and VeRA each appear at a single budget. The paper does not attempt to match the parameter counts across methods — LoRA uses 0.3M while VeRA uses 0.043M for base — making the comparison inherently favorable to VeRA on the parameter efficiency axis. The paper's claim is not "VeRA outperforms LoRA at the same parameter count" for RoBERTa-base, but rather "VeRA achieves competitive performance with far fewer parameters." For RoBERTa-large, the matched-performance-at-different-budgets result is cleaner.

The paper also reports relative performance gain per 1K trainable parameters in Appendix B (Figure 4) for the RTE task with RoBERTa-large. VeRA achieves a gain of 2.64% per 1K parameters, compared to LoRA's 0.18% and the next-best adapter method (AdptP at 3M) achieving 0.05%. This metric dramatically favors methods with very small parameter counts, but it quantifies the efficiency tradeoff clearly: for each trainable parameter added beyond the classification head, VeRA delivers the most accuracy improvement.

E2E Benchmark: Natural Language Generation

Table 3 shows results on the E2E benchmark for GPT-2 Medium and Large. For GPT-2 Medium, VeRA achieves 70.1 BLEU using 98K parameters (0.098M), outperforming LoRA's 68.9 BLEU using 350K parameters (0.35M). This is a 3.6× parameter reduction with better performance. VeRA also leads on NIST (8.81 vs. 8.69), METEOR (46.6 vs. 46.4), and ROUGE-L (71.5 vs. 71.3), while trailing marginally on CIDEr (2.50 vs. 2.51). For GPT-2 Large, VeRA achieves 70.3 BLEU using 170K parameters (0.17M), slightly ahead of LoRA's 70.1 BLEU using 770K parameters (0.77M) — a 4.5× reduction. VeRA leads on NIST (8.85 vs. 8.80), METEOR (46.9 vs. 46.7), and CIDEr (2.54 vs. 2.52), and trails marginally on ROUGE-L (71.6 vs. 71.9).

The E2E results are arguably the paper's strongest: VeRA outperforms LoRA on both model sizes across most metrics while using 3–4.5× fewer parameters. This is a cleaner demonstration of the method's effectiveness than GLUE, where VeRA generally matches rather than exceeds LoRA. The E2E task evaluates open-ended text generation (producing restaurant descriptions from structured meaning representations), which may benefit from the high-rank random basis (VeRA uses r = 1024 for both model sizes) providing diverse expressive capacity that LoRA's learned low-rank basis (rank not specified for E2E in the paper, but typically 4–16 in LoRA implementations) cannot match at the same parameter budget.

A caveat: the E2E results are reported from a single run (last epoch), without error bars or multiple seeds. The absolute differences — 70.1 vs. 68.9 BLEU for Medium, 70.3 vs. 70.1 for Large — are small relative to typical run-to-run variance in NLG tasks, and the paper does not provide the evidence needed to assess statistical significance. The consistency across model sizes and most metrics is suggestive but not conclusive.

Instruction Tuning: Llama 7B and 13B

Table 4 presents MT-Bench scores for instruction-tuned Llama and Llama2 models. The key comparison: for Llama 13B, VeRA achieves 5.22 using 2.4M parameters, while LoRA achieves 5.31 using 250.3M parameters — a 104× reduction for a 0.09-point difference. For Llama2 13B, VeRA achieves 5.93 using 2.4M parameters, actually outperforming LoRA's 5.77 using 250.3M parameters — a 104× reduction with a 0.16-point improvement. For the 7B variants, VeRA scores 4.77 (Llama 7B) and 5.08 (Llama2 7B) vs. LoRA's 5.03 and 5.19 respectively — small gaps of 0.26 and 0.11 points.

The absolute scores provide context: the unadapted Llama 13B model scores only 2.61, so both methods provide substantial improvements (~2.6–2.7 point gain). The fact that VeRA achieves essentially the same gain as LoRA with 100× fewer stored parameters is the paper's most dramatic demonstration of its storage efficiency claim.

Several details matter. First, VeRA uses rank 1024 for instruction tuning (Table 9), compared to LoRA's rank 64 — meaning VeRA's random basis has 16× more directions than LoRA's learned basis. This is enabled by the seed-based regeneration: increasing VeRA's rank from 64 to 1024 adds only 960 entries to each layer's d vector (~4 KB per layer in single precision), while increasing LoRA's rank equivalently would multiply its parameter count by 16×, making it infeasible. Second, VeRA applies adaptation to "all linear layers except the top one" (Section 4.3), a broader application pattern than LoRA's typical query+value adaptation, which partially explains the comparable performance despite the parameter reduction — VeRA makes up for its simpler per-layer adaptation by adapting more layers. Third, the MT-Bench evaluation uses GPT-4 as a judge (the "LLM-as-a-judge" protocol), which introduces its own variance and biases; the paper does not report multiple evaluations or inter-judge agreement.

A curious result: Llama2 13B with VeRA (5.93) outperforms Llama2 13B with LoRA (5.77). The paper does not offer specific analysis for this inversion, but it may reflect the high rank of VeRA's basis (1024) providing more expressive capacity than LoRA's rank-64 learned basis when adapting the larger, more capable Llama2 model. This is consistent with the scaling experiment in Figure 2, where VeRA with high rank outperforms LoRA when parameter budgets are matched — here, the comparison is not budget-matched (VeRA uses far fewer parameters), yet VeRA still wins, suggesting that the rank-64 LoRA may be underfitting the Llama2 13B model.

Appendix F (Table 13) provides an earlier evaluation using Vicuna Eval (the predecessor to MT-Bench) on a 10K subset of the cleaned Alpaca dataset with Llama 7B. VeRA scores 7.48 vs. LoRA's 7.36, again slightly outperforming while using 1.4M vs. 159.9M parameters. The appendix also includes qualitative examples of model outputs with GPT-4 judgments, showing VeRA-generated responses that are comparably detailed and coherent to LoRA-generated responses, though the examples are cherry-picked and not systematically analyzed.

Image Classification: Vision Transformers

Table 5 shows few-shot image classification results (10 samples per class) for ViT-Base and ViT-Large across four datasets. For ViT-Base, VeRA achieves 24.6K parameters vs. LoRA's 294.9K (12× reduction). Performance is comparable on CIFAR100 (84.8 vs. 85.9), Food101 (89.0 vs. 89.9), and RESISC45 (77.0 vs. 77.7), while VeRA outperforms LoRA on Flowers102 (99.0 vs. 98.8). For ViT-Large, VeRA achieves 61.4K parameters vs. LoRA's 786.4K (12.8× reduction). VeRA outperforms LoRA on CIFAR100 (87.5 vs. 87.0), Flowers102 (99.2 vs. 99.1), and RESISC45 (78.6 vs. 78.3), while trailing on Food101 (79.2 vs. 79.5).

The ViT-Large results are particularly noteworthy because VeRA with 12.8× fewer parameters actually achieves better performance than LoRA on three of four datasets. This is consistent with the E2E and Llama2 findings — the shared random basis with high rank (VeRA uses r = 256 for ViT, vs. LoRA's r = 8) may provide more expressive capacity for adapting large vision models than a low-rank learned basis, especially in few-shot settings where the small training set makes it difficult to learn a good low-rank decomposition from scratch.

All methods substantially outperform the head-only baseline (77.7 → 84.8+ for ViT-Base on CIFAR100, 79.4 → 87.0+ for ViT-Large), confirming that the adaptation is providing meaningful task-specific learning beyond just training the classifier. Full finetuning remains the top performer on most dataset-model combinations (e.g., 86.5 vs. VeRA's 84.8 on CIFAR100 with ViT-Base), but the gaps are small relative to the parameter efficiency gain.

Scaling the Number of Trainable Parameters

Figure 2 plots accuracy vs. number of trainable parameters for LoRA and VeRA on the RTE task using RoBERTa-large, sweeping ranks r = {1, 2, 4, 8, 16, 32, 64} for LoRA and r = {1, 4, 16, 64, 256, 1024} for VeRA. Two findings emerge:

VeRA is more parameter-efficient at every operating point. For any given trainable parameter budget, VeRA achieves higher accuracy than LoRA. The gap is small at very low parameter counts (~10⁴) but widens substantially as the budget increases. At ~10⁵ parameters (VeRA r = 1024 vs. LoRA r ≈ 8–16), VeRA outperforms LoRA by approximately 2 accuracy points (roughly 85.5% vs. 83.5%, reading from Figure 2).
VeRA with matched parameters substantially outperforms LoRA. The paper notes that "when the higher-rank VeRA has the same number of parameters as standard LoRA, it outperforms LoRA by 4 accuracy percentage points" (Section 4.5). This refers to the rightmost points: VeRA at r = 1024 (~61K parameters) achieves ~85.5%, while LoRA at its highest rank r = 64 (~800K parameters) achieves ~84% — but VeRA at the same parameter count as LoRA's r = 64 would require an even higher rank (not shown), where the gap would be approximately 4 points based on extrapolation.

This experiment addresses a potential criticism: that VeRA only appears efficient because it uses fewer parameters, and that giving LoRA the same parameter budget would close the gap. The data in Figure 2 refute this — VeRA's parameterization is not just more efficient for a given performance level; it is more effective for a given parameter budget. Even when LoRA is allowed equivalent or greater parameter counts, VeRA's high-rank frozen basis plus learned scaling produces better accuracy than LoRA's learned low-rank basis.

Ablation Studies and Robustness Checks

Single scaling vector vs. both (Table 6a): Removing either b or d from VeRA significantly degrades performance on RoBERTa-large for MRPC and RTE. "Only d" (removing b) achieves 89.7 MRPC / 67.0 RTE vs. full VeRA's 90.5 / 85.8. "Only b" (removing d) achieves 81.6 / 64.3. Both ablations show substantially higher variance (standard deviations of 10–14 on RTE) compared to full VeRA (0.7), indicating that single-vector configurations are unstable — they sometimes train adequately and sometimes fail catastrophically. The paper interprets this as d being more expressive (it modulates the basis before the second projection) while b provides output-level scaling that stabilizes training.

Matrix initialization strategy (Table 6b): Kaiming uniform initialization (the default) achieves 90.5 MRPC / 85.8 RTE, slightly outperforming Kaiming normal (90.0 / 82.6) and dramatically outperforming plain uniform [0, 0.1] initialization (68.9 / 53.1). This confirms that proper variance scaling of the frozen matrices is critical. The poor performance of uniform [0, 0.1] likely reflects the fact that entries are all positive and bounded, limiting the effective rank of the random basis.

Scaling vector initialization (Table 6c): The initial value of d strongly affects performance. d_init = 10⁻¹ achieves 90.5 MRPC / 85.8 RTE, while d_init = 10⁻⁷ achieves 90.8 / 84.7 (comparable), but d_init = 1.0 drops to 70.3 / 60.3 — a catastrophic 20-point degradation on both tasks. The paper hypothesizes that small initial d values allow easier sign changes during optimization (the optimizer can quickly flip small values from positive to negative), providing more flexibility.

Sharing vs. unique random matrices (Table 7): Using unique frozen matrices per layer instead of shared matrices yields negligible differences: MRPC 90.7 ± 0.3 (unique) vs. 90.0 ± 0.9 (shared), RTE 84.6 ± 0.8 vs. 84.6 ± 1.5, CoLA 68.3 ± 1.8 vs. 67.7 ± 0.8, STS-B 91.5 ± 0.2 vs. 91.5 ± 0.6. The overlapping standard deviations mean none of these differences are statistically meaningful. This validates the paper's central design choice that per-layer basis diversity is unnecessary.

Layer-wise adaptation magnitude (Figure 3): Analysis of the learned d vector magnitudes across layers for RoBERTa-large on RTE shows that query matrices consistently receive larger adaptation (higher ||d||₂) than value matrices, and later layers receive larger adaptation than earlier layers, with the magnitude roughly monotonically increasing from layer 0 to layer 23. This pattern — which emerges automatically from training — aligns with prior findings that later transformer layers are more task-specific, and it demonstrates that VeRA's per-layer scaling vectors successfully capture layer-varying adaptation needs despite the shared basis.

Cosine similarity of trained weights (Appendix D, Figure 5): Comparing the effective weight updates ΔW learned by LoRA and VeRA at the same rank (r = 64) on Llama 7B, the average cosine similarity is ~~2 × 10⁻³, approximately 25× higher than the similarity between LoRA weights and random matrices (~~−8 × 10⁻⁵). The similarity is higher in later layers (reaching ~5 × 10⁻³), consistent with the finding that later layers undergo larger adaptation. This suggests that VeRA approximates LoRA-like updates in the later layers where adaptation is most critical, while the low absolute similarity values indicate the two methods are not learning identical weight changes — they find different solutions that achieve similar task performance.

Expressivity comparison (Appendix E, Figure 6): On the synthetic task of fitting random 10×10 matrices, LoRA and VeRA achieve comparable mean squared error (MSE) for any given number of trainable parameters. VeRA can operate in regimes below LoRA's rank-1 (fewer than ~20 parameters for a 10×10 matrix), providing finer-grained control over the parameter budget. This ablation demonstrates that VeRA's parameterization does not inherently limit expressivity — for the same parameter count, it can represent matrices as accurately as LoRA.

Training time and memory (Appendix C, Table 12): At matched rank r = 64 on Llama 7B instruction tuning, VeRA training takes 578 minutes vs. LoRA's 568 minutes (1.8% overhead) but uses 21.69 GB GPU memory vs. 23.42 GB (7.4% reduction). The memory saving arises because VeRA doesn't store optimizer states for the frozen matrices. This is only a partial comparison — in practice, VeRA uses much higher ranks (1024) for its main results, which would increase the training time overhead (more vector multiplications) and the memory advantage (more optimizer states saved). The paper does not provide training cost data for the actual high-rank configurations used in its headline results.

Relative performance gain per 1K parameters (Appendix B, Figure 4): On the RTE task with RoBERTa-large, VeRA achieves a performance gain of 2.64% per 1K trainable parameters relative to a head-only baseline, compared to LoRA's 0.18% and ≤0.17% for all adapter variants. This metric quantifies the efficiency of each method's learned parameters: for every 1,000 additional parameters, VeRA extracts more accuracy improvement than any baseline.

Critical Assessment

Claim 1: VeRA "maintains the same performance" as LoRA while significantly reducing trainable parameters.

The evidence for this claim varies by experimental setting. On RoBERTa-large (GLUE), the claim is well-supported: VeRA achieves an identical 87.8 average to LoRA with 13× fewer parameters (Table 2). On RoBERTa-base, VeRA trails LoRA by 1.4 points (85.2 vs. 86.6) — a non-trivial gap that is particularly acute on RTE (78.7 vs. 86.6, a 7.9-point gap). The paper acknowledges this implicitly by emphasizing the large-model result in its abstract and introduction, but the base-model result qualifies the "same performance" claim: on smaller models, VeRA sacrifices some accuracy for parameter efficiency, and the gap is task-dependent.

On E2E (Table 3), VeRA actually outperforms LoRA on GPT-2 Medium and Large, so the claim holds and is strengthened — VeRA achieves better performance with fewer parameters. On instruction tuning (Table 4), VeRA closely matches LoRA on 7B models (within 0.1–0.3 points on MT-Bench) and matches or exceeds LoRA on 13B models, so the claim is supported, though the single-run nature of MT-Bench evaluation means the small differences (5.22 vs. 5.31) may not be statistically robust. On image classification (Table 5), VeRA matches LoRA on ViT-Base (within 1 point on most datasets) and outperforms LoRA on ViT-Large for three of four datasets, providing additional support.

A critical nuance: VeRA consistently uses much higher ranks than LoRA — r = 1024 vs. typical LoRA r = 8–64 for language tasks — meaning the comparison is between a high-capacity frozen basis + tiny learned modulation and a low-capacity learned basis. It is possible that LoRA with similarly high ranks would perform better, but this is infeasible due to storage constraints, which is precisely VeRA's point. The comparison is fair in the sense that both methods operate at their respective practical operating points, but it does not isolate whether the performance comes from the parameterization or simply from using more basis directions.

Claim 2: VeRA achieves a "ten-fold reduction" in parameters on GLUE for RoBERTa-large.

Strictly supported. RoBERTa-large LoRA uses 0.8M parameters; VeRA uses 0.061M, a ratio of 13.1× (Table 2). The "ten-fold" claim in the abstract and Section 5 is conservative relative to the actual numbers.

Claim 3: VeRA achieves a "100× reduction" on Llama 13B instruction tuning.

Supported but requires context. Llama 13B LoRA uses 250.3M parameters; VeRA uses 2.4M, a ratio of 104× (Table 4). However, the LoRA number (250.3M) is for rank 64 applied broadly to all linear layers as proposed by Dettmers et al. (2023). If LoRA were applied more narrowly (e.g., only query and value layers, as in the GLUE experiments), its parameter count would be lower, and the reduction ratio would be smaller. The 100× figure is specific to the broad application pattern, which is a realistic deployment choice but not inherent to the LoRA method. Additionally, VeRA uses rank 1024 for this comparison, vs. LoRA's rank 64 — so it is achieving comparable performance with 100× fewer stored parameters by using a 16× richer (but regenerated) basis.

Genuine Weaknesses in the Experimental Design

Single-run results on key benchmarks. The E2E, MT-Bench, and image classification results are reported from single training runs without replication across random seeds. This is a significant limitation because: (1) MT-Bench scores from GPT-4 evaluation have known variance, and differences of 0.1–0.3 points (the gaps between VeRA and LoRA on Llama 7B/13B) may not be reliable; (2) E2E NLG metrics like BLEU have run-to-run variance that can exceed the observed differences (70.3 vs. 70.1 for GPT-2 Large); (3) few-shot image classification on 10 samples per class is inherently high-variance, and single-run results may not be representative. The GLUE experiments (5 seeds, median reported) provide a model for what proper statistical protocol looks like; the paper would be strengthened by replicating this rigor on the other benchmarks.

No comparison at matched parameter counts (except Figure 2). The main GLUE, E2E, and instruction-tuning comparisons use different parameter budgets for VeRA and LoRA, making the comparison inherently favorable to VeRA on the efficiency axis. Figure 2 provides the only matched-budget comparison (on RTE only), showing VeRA outperforms LoRA when both use the same number of parameters. This is a strong result, but it is limited to a single task (RTE) and a single model (RoBERTa-large). The paper would be strengthened by matched-budget comparisons on more tasks — for instance, reducing VeRA's rank until its parameter count equals LoRA's on GLUE, and checking whether performance remains competitive. This would test whether the efficiency comes from the parameterization itself or simply from using fewer parameters.

Absence of LoRA results at VeRA-comparable ranks. VeRA uses r = 1024 for many experiments; LoRA is never tested at comparable ranks because it would be infeasibly expensive to store. This is a practical constraint that VeRA exploits, but it leaves open the question: does VeRA perform well because the "frozen basis + learned scaling" parameterization is inherently superior, or because it can afford to use a much richer basis (higher r) at the same storage cost? An experiment where LoRA is trained with r = 1024 and then the matrices are discarded (simulating VeRA's regeneration) would be circular, but an experiment comparing VeRA against LoRA at matched basis dimensionality (e.g., VeRA r = 64 vs. LoRA r = 64) — regardless of storage — would help disentangle the parameterization effect from the rank effect.

The MNLI trick is omitted. The paper notes (Section 4.1) that it omits the "MNLI trick" — initializing RoBERTa-base with weights finetuned on MNLI before training on MRPC, RTE, and STS-B — due to "time constraints and budget limitations." This trick was used in the original LoRA paper and contributed to its reported performance on these tasks. Omitting it makes VeRA's results not directly comparable to LoRA's reported numbers for these tasks, since LoRA benefited from MNLI pretraining while VeRA did not. The paper acknowledges this but does not quantify the expected impact, making the GLUE comparison slightly unfair to VeRA on MRPC, RTE, and STS-B.

No evaluation of the seed-based regeneration in practice. While VeRA's storage advantage is analytically derived (Section 3.2) and quantified in Table 1, the paper never actually measures end-to-end storage, load time, or serving latency in a realistic deployment scenario. The regeneration of matrices from a seed at load time is assumed to have negligible cost, but this is not measured — regenerating a 12,288 × 1024 matrix for every layer of GPT-3 requires non-trivial computation. A practical evaluation measuring: (1) time to load a VeRA-adapted model vs. a LoRA-adapted model, (2) storage requirements on disk for an actual saved checkpoint, and (3) inference throughput after merging — would significantly strengthen the practical claims.

Limited investigation of training stability. The high variance in the single-vector ablations (Table 6a, RTE standard deviation of 13.9 for "only d") suggests that VeRA's training dynamics can be unstable under certain configurations. The paper does not investigate this further — no learning curves, no loss landscape analysis, no discussion of whether the high-rank frozen basis introduces optimization challenges that the default initialization masks. The sensitivity to d_init (Table 6c: 1.0 crashes performance) further suggests that VeRA requires careful hyperparameter tuning, but the paper doesn't quantify how much tuning was needed to achieve the reported results or how sensitive the method is to the hyperparameters that were not swept (optimizer choice, warmup ratio, learning rate schedule).

Missing comparison to full storage-aware baselines. The paper does not compare against methods specifically designed for storage efficiency, such as weight quantization of LoRA matrices (storing A and B in 8-bit or 4-bit precision), pruning of LoRA matrices, or basis-sharing variants of LoRA where per-task B matrices share a global A matrix. These approaches could narrow the storage gap without requiring VeRA's reparameterization. A comparison against LoRA with quantized storage (e.g., storing the 0.8M LoRA parameters in 4-bit = 0.4 MB, compared to VeRA's 0.061M parameters in 16-bit ≈ 0.12 MB) would provide a more nuanced picture of the practical storage advantage.

Domain and architecture coverage. As the paper acknowledges in Section 5, all experiments use Transformer architectures. The approach's effectiveness on convolutional networks, state-space models, or retrieval-augmented architectures is untested. Within Transformers, the paper tests encoder-only (RoBERTa), decoder-only (GPT-2, Llama, Llama2), and vision encoder (ViT) variants, which is reasonable coverage but leaves gaps — no encoder-decoder models (T5, BART), no mixture-of-experts architectures, and no multimodal models. The consistent positive results across the tested architectures are encouraging but do not guarantee universality.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted For

The assumption or constraint. The compute-optimal framework requires estimating each prompt's difficulty before deciding how to allocate the inference budget. The paper's method for doing so — generating 2048 samples per question and averaging PRM final-answer scores across them — is extraordinarily expensive. Section 3.2 acknowledges this directly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The paper frames this as an "exploration-exploitation tradeoff" — compute spent assessing difficulty versus compute spent solving the problem — and flags it as "a key avenue for future work."

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. Generating 2048 samples per question to estimate difficulty consumes far more compute than the largest test-time budgets studied (256–512 generations). If this cost were included, the "compute-optimal" strategy would likely underperform a simple best-of-N baseline at most practical budgets, because the difficulty estimation overhead would consume the very savings the adaptive allocation claims to provide. The 4× figure should therefore be understood as an upper bound on achievable efficiency contingent on a cheap difficulty estimator that does not yet exist.

What evidence exists in the paper. The paper provides no measurement of the amortized cost. The difficulty estimation cost is stated qualitatively in Section 3.2 but never quantified in generations, FLOPs, or wall-clock time, and it never appears in any budget calculation, figure, or table. The paper does not report how long difficulty estimation takes relative to strategy execution, nor does it explore cheaper estimation methods (e.g., using fewer than 2048 samples, training a lightweight classifier to predict difficulty from the prompt text alone).

Mitigation status. Not addressed. The paper explicitly defers this to future work: "we leave the exploration of more efficient difficulty estimation methods to future work." No experiments test whether the predicted difficulty bins remain effective with substantially fewer samples (e.g., 16 or 64 rather than 2048). This is the single largest gap between the paper's analytical framework and its practical deployability.

Hard Problems Remain Fundamentally Unsolved by Test-Time Compute

The assumption or constraint. The compute-optimal framework presupposes that the base model can generate correct solutions at some non-trivial rate. When this fails — when pass@1 is near zero — no amount of search or revision can help. The paper's difficulty bin 5 (hardest questions, lowest pass@1) isolates this regime.

The consequence. Test-time compute does not create new capabilities; it only amplifies existing ones. For problems fundamentally outside the base model's reach, the approach provides essentially zero benefit regardless of budget. This is not a failure of the allocation strategy — it is a hard boundary on what test-time compute can achieve. Practitioners with problem distributions skewed toward genuinely hard or out-of-distribution tasks should not expect test-time compute to substitute for pretraining.

What evidence exists in the paper. The evidence is stark and consistent across all methods. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, and all three stars (representing the ~14× larger model at different R values) sit above the scaling curves. Section 7's takeaway box explicitly states: "On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated."

Mitigation status. The paper is transparent about this limitation — it is described as a finding, not a weakness. No mitigation is attempted or proposed because the limitation is fundamental: "test-time compute can amplify existing capability but does not create it from nothing." The paper's contribution is precisely characterizing this boundary, not transcending it. For practitioners, the practical mitigation is to route hard problems to a larger model rather than investing test-time compute in the smaller one.

The FLOPs-Matched Baseline Is Not Compute-Optimal, Making the Pretraining vs. Inference Comparison Favorable to Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). Section 7 acknowledges:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the ~14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search of its own.

The consequence. A Chinchilla-optimal model (Hoffmann et al., 2022) trained with ~14× more total FLOPs — scaling both parameters and data optimally — would likely outperform a parameter-only-scaled model. This makes the pretraining baseline weaker than it could be, and the reported advantages of test-time compute over pretraining (e.g., +27.8% on medium problems at R ≪ 1 for revisions) may shrink or reverse against a properly compute-optimal larger model. Similarly, giving the larger model even a modest test-time compute budget (say, best-of-8 or a short revision chain) would create a substantially stronger baseline. The current comparison answers "test-time compute on a small model vs. greedy decoding on a larger model" — a narrower question than "test-time compute vs. pretraining compute" generally.

What evidence exists in the paper. The paper is explicit about this design choice in Section 7 but provides no sensitivity analysis — no results with a Chinchilla-optimal larger model, and no results where the larger model receives any test-time compute. The comparison is entirely between PaLM 2-S* with compute-optimal test-time scaling and a single larger model with greedy decoding. The paper acknowledges the limitation verbally but does not quantify its impact.

Mitigation status. Partially addressed through transparency. The caveat is stated clearly, and the paper frames the comparison as "representative of a canonical approach to scaling pretraining compute" rather than as the strongest possible baseline. The explicit call for future work on "compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally" signals awareness of the gap. However, the headline claim — that test-time compute with a smaller model can outperform a ~14× larger model — should be interpreted as conditional on the specific pretraining recipe used for the larger model.

Results Are Demonstrated on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. All experiments use the MATH benchmark (500 test questions, high-school competition-level math) with PaLM 2-S* as the base model. The paper states in Section 4 that it "believes this model is representative of the capabilities of many contemporary LLMs" but provides no evidence for this representativeness across model families or reasoning domains.

The consequence. It is unknown whether the paper's central findings — the difficulty-dependent optimal strategies, the 4× efficiency gains, the specific over-optimization thresholds — generalize to other reasoning domains (code generation, logical deduction, scientific QA), other model families (GPT, Claude, Gemini, open-source alternatives), or tasks requiring factual recall rather than multi-step inference. MATH consists exclusively of symbolic math problems with unambiguous ground-truth answers, which enables both the PRM training pipeline (Monte Carlo rollout supervision requires correctness checking) and the difficulty estimation method. Tasks without clean verifiability (open-ended generation, creative writing, complex planning) may not support the same framework at all.

What evidence exists in the paper. None beyond MATH and PaLM 2-S*. The paper does not include experiments on other reasoning benchmarks (e.g., GSM8K for grade-school math, ARC for science reasoning, HumanEval for code), other model families, or non-reasoning tasks. The 500-question test set, split into five difficulty quintiles of ~100 questions each and further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin — a small sample that may not produce robust strategy selection.

Mitigation status. Not addressed. The paper does not claim generality beyond MATH and PaLM 2-S*, but it also does not test the boundaries. Section 8 does not explicitly call for multi-benchmark or multi-model replication. A practitioner cannot assume from this paper that the difficulty-dependent allocation patterns (beam search on medium, best-of-N on easy, etc.) will transfer to their domain or model without independent verification.

The Revision Model Exhibits a 38% Correct-to-Incorrect Reversion Rate, and Revision Training Is Fragile

The assumption or constraint. The revision model is trained only on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). At test time, it may encounter correct answers in its own context (produced during earlier revision steps) and incorrectly "revise" them into wrong answers. The paper reports that approximately 38% of correct answers get converted back to incorrect ones using a naive approach. The paper mitigates this by selecting the best answer from anywhere in the revision chain rather than always taking the final revision, but this is a post-hoc patch rather than a solution.

The consequence. The revision model is unreliable as an iterative improver — it can and does degrade already-correct answers. This limits the effectiveness of long revision chains, since the probability of regressing from a correct answer accumulates with chain length. It also means the revision model cannot be used as a standalone "self-improver" without an external verifier or selection mechanism to catch regressions. The paper's sequential revision strategy (Section 6) works despite this reversion problem because it uses majority voting or verifier-based selection to pick the best answer across the entire chain, but this adds complexity and requires storing all intermediate revisions.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The ReST^EM experiment (Appendix K, Figure 16) provides additional evidence of training fragility: attempting to further optimize the revision model with RL-style training caused sequential revisions to substantially hurt performance (dropping from ~38.5% at the optimal ratio to ~33.5% with fully sequential at 256 generations). The paper hypothesizes that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This negative result demonstrates that the revision approach is sensitive to training methodology in ways that are not fully characterized.

Mitigation status. Partially mitigated through answer selection across the revision chain (majority voting or verifier-based selection), but this is a workaround, not a fix. The paper does not explore training the revision model to recognize when no revision is needed (e.g., by including correct-to-correct trajectories in training data), nor does it investigate architectural solutions (e.g., a confidence threshold below which revision is skipped). The reversion problem and the ReST^EM failure together suggest that revision-based test-time compute is more brittle than the aggregate results suggest, and positive outcomes depend on specific training choices (offline data construction, edit-distance-based pairing) that may not transfer to other settings.

Sequential Revisions Introduce Latency That Is Not Accounted For

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial — each revision depends on the previous one — while parallel best-of-N can be executed simultaneously given sufficient hardware. The paper does not discuss this tradeoff.

The consequence. In latency-sensitive applications (interactive assistants, real-time decision-making systems), the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. A strategy allocating 128 generations as 64 sequential × 2 parallel requires roughly 64× longer wall-clock time than running 128 parallel samples simultaneously. For a user waiting for a response, a 64-step sequential revision chain (each step requiring a full autoregressive generation) would be prohibitively slow even if the total FLOPs are modest. The compute-optimal framework optimizes for FLOPs efficiency, not latency, which are fundamentally different objectives when sequential dependencies exist.

What evidence exists in the paper. None. The paper never measures wall-clock time, never discusses latency implications, and never compares sequential vs. parallel strategies in terms of response time. The "generation" abstraction — treating a sequential chain of N revisions as equivalent in cost to N parallel samples — is valid for total FLOP accounting but collapses the latency dimension entirely.

Mitigation status. Not addressed. The paper does not acknowledge latency as a concern, propose latency-aware allocation strategies, or discuss the tradeoff between throughput (total FLOPs) and latency (wall-clock time). A practitioner deploying the sequential revision strategy in a user-facing application would discover this limitation immediately upon measuring response times. Future work on latency-aware compute-optimal allocation — perhaps preferring parallel strategies when latency matters and sequential strategies only for batch processing — would be a natural extension.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing of parameter-efficient finetuning that shifts the field's understanding of what must be stored versus what can be regenerated. Rather than treating adaptation as "learning a low-rank subspace" (the LoRA paradigm), VeRA recasts it as "learning scaling coefficients on a fixed random basis shared across all layers." This is not an incremental compression of LoRA — it is a fundamentally different decomposition of the adaptation problem with qualitatively different storage scaling properties.

The magnitude of this shift is best understood through its economic implications rather than any single accuracy number. The paper demonstrates that a single pair of random matrices, when modulated by per-layer vectors totaling a few megabytes, can support adaptation across language understanding, language generation, instruction following, and visual recognition — tasks that prior work implicitly assumed required task-specific learned subspaces. The fact that VeRA achieves this while matching LoRA's performance on RoBERTa-large (87.8 average on GLUE) and outperforming LoRA on GPT-2 Medium for E2E (70.1 vs. 68.9 BLEU) — all while using 3× to 100× fewer stored parameters — establishes that most of what we thought of as "task-specific basis discovery" is actually task-specific scaling of a universal adaptation basis. The basis itself carries negligible task-specific information; the scaling vectors encode the adaptation.

This finding should redirect research attention from "how do we learn better low-rank subspaces?" toward "how rich does a random basis need to be, and what modulation mechanisms best exploit it?" Methods like AdaLoRA (Zhang et al., 2023b), which invest substantial complexity in dynamically reallocating the learned basis across layers, may be solving a problem — basis discovery — that VeRA's results suggest is largely unnecessary when the basis is sufficiently high-dimensional and shared. The paper's evidence that VeRA with r = 1024 outperforms LoRA even when parameter budgets are matched (Figure 2, ~4 percentage point gap on RTE) directly supports this: given enough random directions, learned scaling outperforms learned directions at the same storage cost.

The paper also redefines what "parameter efficiency" means for deployment. Prior PEFT work measured efficiency by trainable parameter count during finetuning, implicitly assuming that training cost was the bottleneck. VeRA shifts the metric to stored parameters per adapted model, treating storage — not training FLOPs — as the binding constraint for personalization at scale. This reframing is grounded in the paper's motivating calculation: one million LoRA-adapted GPT-3 models require 275 TB of storage, while VeRA-adapted equivalents would require roughly 10 TB. The paper's 1.8% training time overhead (Table 12, 578 vs. 568 minutes for Llama 7B at matched rank) is a small price to pay for a 7.4% training memory reduction and orders-of-magnitude storage reduction at deployment. This recasts the efficiency tradeoff: slightly slower training for dramatically cheaper serving is the right bargain when the deployment scenario involves many adapted models.

The paper also reconciles a latent tension in the intrinsic dimensionality literature. Aghajanyan et al. (2021) showed that finetuning has low intrinsic dimension (d₉₀ ≈ 896 for RoBERTa-base), yet LoRA used 300K parameters — a ~300× gap. VeRA closes much of this gap, achieving competitive performance with 43K parameters on RoBERTa-base, suggesting that the earlier intrinsic dimension estimates were not lower bounds on what was achievable but rather reflections of LoRA's parameterization overhead. VeRA demonstrates that getting closer to the intrinsic dimension requires decoupling basis capacity from storage cost — using a high-dimensional basis (r = 1024) to provide expressivity while storing only the low-dimensional modulation (the scaling vectors). This principle — "store the modulation, regenerate the basis" — is broadly applicable beyond VeRA's specific formulation and represents a design pattern that subsequent PEFT methods can adopt.

Concretely, the paper makes several research directions more attractive:

Frozen basis methods across architectures: VeRA's success on Transformers invites exploration of similar parameterizations for CNNs, state-space models, and mixture-of-experts architectures.
Learned modulation mechanisms beyond diagonal scaling: If two diagonal vectors can achieve this much, what about low-rank modulation, attention-based modulation, or conditional modulation based on input?
Dynamic basis selection: The paper shows that a single random basis works across all layers; could the system learn to select different bases for different layer types or different tasks?

And it makes some directions less attractive:

Per-layer learned low-rank decompositions: If sharing a frozen basis across all layers works this well, the case for per-layer learned bases weakens substantially.
Sophisticated budget allocation for learned bases: AdaLoRA's dynamic pruning may be solving a problem that can be bypassed entirely by using a frozen overcomplete basis and learned scaling.

Follow-Up Research This Work Enables

Cheap difficulty estimation for compute-optimal allocation. The most immediate bottleneck the paper identifies (Section 3.2) is the cost of estimating question difficulty — 2048 samples per question is far too expensive for deployment. A natural follow-up would train a lightweight difficulty classifier that takes only the question text as input and predicts the difficulty bin. This classifier could be distilled from the PRM's difficulty estimates: use the PRM to compute predicted difficulty bins on a large corpus of questions, then train a small model (e.g., a distilled BERT variant or even a linear classifier on top of the base model's embeddings) to predict those bins from the text alone. The key metric would be whether the classifier's bin assignments produce compute-optimal scaling curves that overlap with the oracle and PRM-predicted curves in Figures 4 and 8. A strong result would show that a classifier with negligible cost (a single forward pass) achieves the same 4× efficiency gain over best-of-N as the expensive PRM-based difficulty estimation. A negative result — that cheap difficulty estimation substantially degrades the compute-optimal policy — would reveal that the PRM's score distribution captures difficulty-relevant information beyond what the question text alone provides, and would motivate research into intermediate approaches (e.g., using 16 rather than 2048 samples for difficulty estimation, or adaptive sampling that stops early when difficulty is confidently estimated).

Combining PRM tree-search with the revision model as the proposal distribution. The paper studies search against the PRM verifier and iterative revisions as independent mechanisms, but explicitly notes they were never combined (Section 8). The natural extension is to use the revision model as the proposal distribution within beam search: at each step of the search tree, instead of generating candidate completions from the base model, generate them from the revision model conditioned on the partial solution and any previous rejected branches. This would combine the revision model's ability to produce higher-quality candidates (better proposal distribution) with the PRM's ability to guide which candidates to explore further (better selection). The experiment would compare three configurations on the MATH benchmark at matched generation budgets: (1) beam search with the base model as proposer (the current Section 5.2 results), (2) the revision model with best-of-N weighted selection (the current Section 6 results), and (3) beam search with the revision model as proposer. The key question is whether the combined system exceeds both individual approaches, particularly on medium-difficulty questions (bins 3–4) where both mechanisms individually show complementary strengths. A strong positive result would demonstrate that the proposal distribution and verifier are multiplicative rather than additive in their benefits — that better candidates are more valuable when you can search among them intelligently. A null result (combined performance ≈ max of individual performances) would suggest that the two mechanisms exploit the same underlying capability improvements and are largely redundant.

Robust PRM training resistant to over-optimization under aggressive search. The paper documents that beam search can degrade performance at high budgets due to PRM over-optimization — particularly on easy problems where the verifier is already mostly correct (Figure 3, right). This is directly analogous to reward hacking in RLHF and represents a fundamental bottleneck for scaling test-time compute. A concrete follow-up would train a PRM with adversarial data augmentation: generate solutions using beam search (not just i.i.d. sampling) and include these search-optimized solutions in the PRM training set with their ground-truth correctness labels. The hypothesis is that exposing the PRM during training to the kind of solutions it will evaluate during deployment — including adversarial ones that exploit its weaknesses — will produce a more robust verifier whose accuracy degrades less under aggressive search. The experiment would train two PRMs on the same base model: one with the standard i.i.d. Monte Carlo rollout data (the paper's current approach), and one augmented with beam-search-generated solutions. Both would be evaluated by measuring the correlation between PRM scores and actual correctness as a function of search depth and budget, and by plotting accuracy vs. generation budget for beam search with each PRM. A successful result would show that the adversarially trained PRM maintains or improves accuracy at high budgets where the standard PRM's performance declines. A negative result — that adversarial training doesn't help or hurts — would suggest that over-optimization is inherent to the PRM's inductive biases and that mitigation requires fundamentally different approaches (e.g., constrained search with KL penalties, ensemble verification, or abstention when the PRM is uncertain).

Dynamic, online difficulty estimation and strategy switching. The paper's difficulty estimation is static and performed once before strategy execution. A more ambitious extension would implement dynamic difficulty estimation: begin with a small number of parallel samples (e.g., 4–8), use the verifier's scores on those initial samples to estimate difficulty in real time, and then allocate the remaining budget accordingly. This amortizes difficulty estimation into the problem-solving process and enables mid-solution strategy switching — for instance, if early samples suggest the problem is easy, switch to sequential revisions; if they suggest it's hard, broaden to parallel search. This approach connects to the multi-armed bandit and Bayesian optimization literatures: the initial samples serve as exploration that informs the exploitation strategy for the remaining budget. The experiment would compare the static compute-optimal policy (with difficulty pre-estimated from 2048 samples, as in the paper) against a dynamic policy with the same total budget, where the first K samples are used for both difficulty estimation and as part of the solution attempt. The key metric is whether the dynamic policy approaches the static policy's performance curve despite amortizing difficulty estimation. A strong result would show that the dynamic policy achieves ~95% of the static policy's accuracy at each budget level, effectively eliminating the need for separate difficulty estimation. A negative result — that dynamic estimation significantly underperforms — would reveal that reliable difficulty estimation requires more samples than can be spared from the solution budget, and would motivate investment in the classifier-based approach described above.

Replication across model families and reasoning domains. All the paper's results are on MATH with PaLM 2-S*. A critical replication study would test whether the paper's central findings — the difficulty-dependent optimal strategies, the 4× efficiency gains, the beam search over-optimization pattern — transfer to substantially different settings. The minimum viable replication would include: (1) a different model family (e.g., Llama 2 or Mistral at comparable scale, tested on MATH to isolate model-family effects), (2) a different reasoning benchmark (e.g., GSM8K for grade-school math, ARC for science reasoning, or HumanEval for code generation, all tested with a single model to isolate domain effects), and (3) measurement of whether the difficulty quintile boundaries shift substantially — do "easy" questions for PaLM 2-S* correspond to "easy" questions for Llama 2, or is the difficulty distribution model-specific? The key finding would be whether the qualitative pattern (beam search for medium, best-of-N for easy, nothing helps on hard) is universal or model/domain-specific. A finding of universality would dramatically increase confidence in the compute-optimal framework's deployability; a finding of model-specificity would reveal that the optimal policy must be tuned per-model-family, increasing the practical barrier to adoption but not invalidating the framework.

Self-improvement loops using compute-optimal test-time strategies for data generation. The paper explicitly envisions (Section 8) using compute-optimal test-time strategies to generate high-quality solutions on training data, then fine-tuning the base model on those solutions, in an iterative loop. A concrete study would implement one iteration of this: take a base model, use compute-optimal test-time strategies (search + revisions, with predicted difficulty bins) to generate solutions on a math training set (e.g., the MATH training split), filter to solutions where the verifier assigns high confidence, and fine-tune the base model on these high-confidence solutions. The evaluation would measure whether the fine-tuned model (with no test-time compute) outperforms the original base model with compute-optimal test-time compute at matched total FLOPs (training + inference). This tests whether test-time compute can be "distilled" into model weights, amortizing the inference cost across many future queries. The paper's negative result with ReST^EM (Appendix K, Figure 16) — where RL-style optimization of the revision model caused performance degradation — indicates that naïve self-improvement can backfire, making the specific data generation and filtering strategy critical. A successful result would demonstrate a virtuous cycle where test-time compute generates better training data, which improves the base model, which in turn benefits more from test-time compute. A failure would reveal that the gains from test-time compute are inherently inference-time phenomena that cannot be compressed into model weights with current techniques.

Practical Applications and Downstream Use Cases

Multi-user personalization for cloud-based AI assistants. This is the application that opens the paper (Section 1) and for which VeRA is explicitly designed. A cloud service provider deploys a large base model (e.g., Llama 13B or GPT-3-scale) and maintains per-user adaptations that personalize the model's behavior — writing style, factual knowledge about the user, task preferences, interaction patterns. With LoRA (rank 64), each user's adaptation requires storing ~250M parameters (~1 GB in single precision, or ~250 MB at 8-bit). With VeRA (r = 1024), each user's adaptation requires storing ~2.4M parameters (~10 MB in single precision). For one million users, the storage requirement drops from ~1 PB to ~10 TB — the difference between needing a distributed storage cluster and fitting on a single server's attached SSDs. The paper's results on instruction tuning (Table 4) provide the key evidence: VeRA achieves 5.22 on MT-Bench vs. LoRA's 5.31, a negligible quality difference (0.09 points on a 10-point scale) for a 100× storage reduction. In this deployment scenario, the base model and the single pair of frozen random matrices are stored once on each GPU server; when a user's request arrives, the system regenerates the shared matrices from the global seed, loads the user's tiny d and b vectors, constructs W_adapted = W_0 + Λ_b B Λ_d A on-the-fly, and serves the request. The per-user loading cost is dominated by the matrix reconstruction (m × n operations per adapted layer), which is a one-time cost amortized over the user's session. The key practical metric not measured by the paper is model swap latency — the time from receiving a user request to having the adapted weights loaded and ready — which would determine whether this architecture is viable for interactive serving or only for batch processing.

On-device deployment of task-specific models with limited storage. Edge devices (phones, laptops, embedded systems) face extreme storage constraints but may need multiple task-specific adaptations of a single base model — for example, a keyboard app that uses a language model fine-tuned for email composition, text messaging, and search queries. Storing three separate LoRA adaptations of a small language model (e.g., GPT-2 with LoRA rank 16, ~0.35M parameters or ~1.4 MB each) costs ~4.2 MB. With VeRA, the same three adaptations at comparable quality (based on the E2E results in Table 3, where VeRA matches or exceeds LoRA for GPT-2 Medium and Large) would cost ~0.3 MB total (assuming ~0.1M VeRA parameters per task, as in the E2E Medium results). The 14× storage reduction matters for devices with 32–64 GB total storage where app size is tightly constrained. Beyond storage, the training memory advantage (Table 12, 7.4% reduction for Llama 7B) is also relevant for on-device finetuning, where GPU memory is severely limited. The paper's image classification results (Table 5, 10 samples per class) are particularly relevant here — they demonstrate that VeRA works in few-shot personalization settings where the device sees only a handful of user examples and must adapt quickly with minimal storage overhead. A practical deployment would store the base model and the frozen matrices in the app binary, perform on-device finetuning using the few-shot examples to learn the user's d and b vectors, and store only those vectors persistently.

Cost-efficient batch inference for organizations running many specialized models. A company maintaining separate fine-tuned models for sentiment analysis, named entity recognition, question answering, summarization, and code generation — all derived from the same base model — faces storage costs proportional to the number of tasks times the adaptation size. With LoRA, six task adaptations of a Llama 7B model (159.9M parameters each at rank 64, Table 4) require ~960M stored parameters (~3.8 GB in single precision). With VeRA, six task adaptations (1.6M parameters each at r = 1024) require ~9.6M stored parameters (~38 MB). For a deployment serving these models from GPU memory, VeRA could fit all six adaptations in GPU memory simultaneously (eliminating model swapping latency between tasks), while LoRA might only fit one or two, forcing costly reloads. The paper's GLUE results (Table 2) demonstrate that this storage reduction comes at minimal quality cost — VeRA achieves identical average performance to LoRA on RoBERTa-large (87.8 vs. 87.8) — and the E2E results (Table 3) demonstrate that VeRA can even outperform LoRA in quality while reducing storage. The training memory reduction (Table 12) also means that these task-specific adaptations can be produced on cheaper hardware, lowering the barrier to entry for organizations without access to high-memory GPUs.

Efficient fine-tuning for large-scale federated learning or continuous personalization. In federated learning scenarios, model updates are communicated from edge devices to a central server. The communication cost is proportional to the update size. VeRA's tiny update size (1.6M parameters for a 7B model, vs. 159.9M for LoRA) means per-round communication is 100× cheaper, making frequent personalization updates feasible over bandwidth-constrained connections. Similarly, for continuous personalization systems that update per-user models daily or hourly, the cost of storing and transmitting VeRA vectors versus LoRA matrices compounds over time. This application is speculative — the paper does not evaluate VeRA in a federated learning setting — but the parameter count comparison in Table 1 and the storage advantage in Table 4 make the efficiency argument directly.

When to Prefer This Method

The paper articulates a clear tradeoff between VeRA and LoRA that centers on the deployment scenario's storage requirements, not on training efficiency or absolute performance. The decision rule is:

Prefer VeRA when: the deployment requires maintaining many adapted versions of a single base model (per-user personalization, multi-task serving with frequent task switching, edge deployment with storage constraints, or federated learning with bandwidth constraints). The paper's motivating calculation (275 TB for one million LoRA-adapted GPT-3 models) and the parameter count formulas (Section 3.2) provide the quantitative basis. VeRA's 3× to 100× storage reduction at competitive or better performance (Tables 2, 3, 4, 5) directly addresses this scenario. The method is particularly well-suited when the adapted models must be loaded into GPU memory on demand — the smaller storage footprint means more adaptations can reside in memory simultaneously, eliminating swap latency.
Prefer LoRA when: the deployment involves only a handful of adapted models (single-task finetuning for research, a few production tasks), storage is not a constraint, and training simplicity is prioritized. LoRA has a more mature ecosystem (HuggingFace PEFT integration, extensive community documentation, established hyperparameter recipes from Hu et al., 2022) and does not require tuning the additional hyperparameters VeRA introduces (the initial value of d, the rank r for the frozen basis vs. the learned basis). The paper's results show LoRA modestly outperforming VeRA on RoBERTa-base (86.6 vs. 85.2 average on GLUE, Table 2), suggesting that for smaller models where the performance gap is measurable, LoRA may be preferred when storage is not limiting. Additionally, the paper's training time comparison (Table 12) shows VeRA is 1.8% slower per step — negligible for most use cases but potentially relevant for very large-scale training where every percentage point matters.

The paper does not provide evidence for choosing between VeRA and other PEFT methods (adapters, BitFit, AdaLoRA) on a performance basis — the comparisons are primarily to LoRA. The decision between VeRA and adapter-based methods additionally involves the inference latency tradeoff: both VeRA and LoRA can merge into pretrained weights for zero-cost inference, while adapters add per-layer computation that cannot be merged away. This gives VeRA and LoRA a structural advantage over adapters for latency-sensitive serving regardless of parameter count.