Scalable Neural Network Kernels

ArXiv: 2310.13225

🎯 Pitch

Neural network feedforward layers can be entirely replaced by kernel operations that disentangle inputs from weights, yielding up to 5× parameter compression with no accuracy loss. The extreme case reveals that a fully bundled network has a closed-form optimal solution under mean squared error, bypassing backpropagation entirely. This reframes deep learning as a scalable kernel method where expressiveness strictly exceeds that of standard dot-product layers.

---

1. Executive Summary

This paper introduces scalable neural network kernels (SNNKs), a new computational module designed to replace standard feedforward layers in neural networks by disentangling the input from the learnable parameters and reconnecting them only through a dot-product kernel at the final computation stage. The authors develop the universal random features (URFs) mechanism to instantiate SNNK variants—leveraging Fourier analysis and softmax-kernel linearization to construct unbiased estimators of activation functions—and further propose the neural network bundling process that applies SNNKs iteratively to compactify deep architectures, with the extreme case yielding a fully bundled network whose optimal parameters admit explicit formulae under losses such as mean squared error. Across pointwise kernel estimation, SIREN-based image fitting, and adapter-based fine-tuning of Transformers on GLUE and vision benchmarks, SNNK-based methods achieve up to a ~5× reduction in trainable parameters while maintaining competitive accuracy, establishing that feedforward computation can be substantially compressed without loss of expressiveness only when the disentangled two-tower representation is coupled with appropriate random feature maps—and further showing that SNNKs are strictly more expressive than regular FFLs, as they can model relationships beyond pointwise transformations of parameter-input dot-products.

2. Context and Motivation

The Core Problem: Feedforward Layers Are Computationally Expensive and Computationally Entangled

The fundamental problem this paper addresses is deceptively simple: standard feedforward layers (FFLs) in neural networks mix inputs and parameters in a way that prevents both computational efficiency and parameter compression. An FFL, as defined in Equation 1, computes:

$x \rightarrow f(Wx + b)$

where $x \in \mathbb{R}^d$, $W \in \mathbb{R}^{l \times d}$, $b \in \mathbb{R}^l$, and $f$ is an activation function applied element-wise. This formulation seems innocuous, but it creates two tightly coupled problems that the paper identifies and tackles:

1. Computational entanglement: The input $x$ and the parameters $(W, b)$ are bound together inside the activation function $f$ through the matrix-vector product $Wx + b$. There is no way to precompute anything about the parameters independently of the input, or vice versa. Every forward pass requires the full $O(ld)$ matrix multiplication.

2. Parameter inefficiency: The weight matrix $W$ contains $l \times d$ trainable parameters. For modern architectures where both $l$ and $d$ are large (e.g., the feedforward blocks in Transformers where the intermediate dimension is $4d$), this dominates the parameter count. Yet there is no obvious way to compress this matrix without losing expressiveness, because the parameters are directly multiplied with the input.

The paper's key insight is that these two problems share a common root cause—the entanglement of inputs and parameters—and can be addressed simultaneously by rewriting the FFL in a "two-tower" kernel formulation where inputs and parameters are transformed independently before being combined via a dot product.

Why This Problem Matters: The Scaling Imperative

The motivation for solving this problem extends beyond academic curiosity into several pressing practical concerns:

The dominance of feedforward computation in large models. In Transformer architectures, the feedforward network (FFN) blocks—which are stacks of FFLs—account for more than half of the total parameters and computation. The paper notes in Appendix J.4 that for BERT-base (Devlin et al., 2019), the FFN blocks alone constitute the majority of the 110M parameters. For larger models, this fraction grows. Any technique that can reduce FFN parameter count or computational cost without sacrificing accuracy therefore has immediate, multiplicative impact on model efficiency.

Deployment constraints in resource-limited settings. As models grow, the gap between what can be trained in datacenters and what can be deployed on edge devices widens. The paper frames SNNKs specifically as a compression mechanism—by disentangling inputs from parameters, the parameter tower $\Psi_f(w, b)$ can be replaced with directly learned vectors of dimension $m \ll d$, reducing the parameter count from $O(ld)$ to $O(lm)$. For $m = 16$ and $d = 768$ (typical BERT hidden size), this is roughly a $48\times$ reduction per layer. This matters for on-device deployment, where memory and compute budgets are fixed and small.

The orthogonal nature of the approach. The paper explicitly positions SNNK compression as orthogonal to existing compression methods like pruning (Liang et al., 2021), distillation (Gou et al., 2021), and quantization (Gholami et al., 2021). These methods operate on already-trained networks by removing weights, transferring knowledge to smaller architectures, or reducing numerical precision. SNNKs, in contrast, replace the computational primitive itself with a more efficient one that can be trained from scratch. The techniques are complementary—one could prune and quantize an already SNNK-compressed network—meaning SNNKs expand the Pareto frontier of what's achievable in model efficiency rather than competing for a fixed point on it.

A bridge between kernel methods and deep learning. Beyond immediate practical gains, the paper addresses a theoretical gap that has persisted since the deep learning revolution. Kernel methods (Schölkopf & Smola, 2002) offer principled guarantees—convex optimization, closed-form solutions, representer theorems—but scale poorly with dataset size ($O(N^3)$ for kernel regression). Neural networks scale well but lack these guarantees. The random features approach of Rahimi & Recht (2007) showed that kernels can be approximated in linear time by mapping inputs to finite-dimensional feature spaces where dot products approximate kernel evaluations. The paper extends this bridge in the opposite direction: can we reinterpret neural network layers as kernel evaluations and thereby bring kernel-method properties to deep architectures? The answer, as developed in Sections 3.1–3.3, is yes—and doing so naturally yields compression and computational savings.

Where Prior Approaches Fall Short

The paper identifies specific limitations in three lines of prior work that it aims to address:

1. Random Feature Methods: Restricted to Positive Definite Kernels and Simple Functions

The literature on random features (Rahimi & Recht, 2007; Choromanski et al., 2018; Kar & Karnick, 2012; Wacker et al., 2022a;b) has made substantial progress in approximating kernels efficiently, but with restrictions that make the methods inapplicable to general feedforward layers:

• Positivity assumptions on kernel functions: Most RF methods assume the target kernel $K(x, y) = f(x^\top y)$ is positive semi-definite. This is a natural requirement in classical kernel methods where the kernel defines a valid reproducing kernel Hilbert space (RKHS). However, as the paper emphasizes in Section 1: "function $K_f$ does not need to be positive semi-definite. This is critical for applications in neural networks, where the activation function $f$ usually does not correspond to a positive semi-definite kernel." Common activations like ReLU, tanh, and sigmoid do not produce positive semi-definite kernels when composed with dot products. The paper's URF mechanism explicitly handles this by decomposing the Fourier transform into positive and negative components (the four distributions $P_0, P_1, P_2, P_3$ in Equation 6).

• Restrictive assumptions on input domains: Some methods require inputs to lie on the unit sphere (Scetbon & Harchaoui, 2021; Han et al., 2022) or assume activation functions with positive Taylor coefficients (Kar & Karnick, 2012). The paper notes that both assumptions are "unrealistic for the neural network applications as far as inputs $x$ are concerned"—real neural network inputs are unbounded and activations like tanh have Taylor series with alternating signs. The paper's attempt to use polynomial kernel methods with tanh (Appendix L.1) confirms this: "applying a sparse kernel distorts [the pretrained features]" and produces considerably worse results.

• Focus on kernel approximation rather than layer replacement: Prior RF work focuses on approximating a given kernel function with random features—the goal is to compute kernel evaluations cheaply. The paper's goal is different: to reinterpret an existing neural network layer as computing expected kernel evaluations and then instantiate it with random features. This shifts the objective from "approximate this kernel well" to "make this layer computationally efficient while preserving its function," which allows for different design choices (e.g., directly learning the random feature vectors rather than applying a fixed transformation $\Psi_f$).

2. Neural Network Linearization: Limited to Shallow Networks or Specific Regimes

A substantial literature has explored the connection between neural networks and kernels, but with limitations that prevent application to general deep architectures:

• Two-layer network restriction: Most work on linearizing neural networks via kernels (Cho & Saul, 2009; 2011; Ghorbani et al., 2020) focuses on the specific architecture:

$J(x; \theta) = \sum_{i=1}^N a_i f(x^\top w_i)$

where $w_i$ and $f$ are fixed and only the top-layer weights $a_i$ are trainable. Under various assumptions, this can be rewritten as a linear model in a feature space defined by the random projections $w_i$. The paper notes (Section 2): "Even though our setting is fundamentally different, as our goal is to linearize single layers to disentangle the weights and the inputs, we build on the above intuition to create our SNNK-layers."

The key difference is that the paper's approach (1) applies to individual layers within deep networks, (2) treats both the layer's weights and the layer's input as variables to be disentangled, not just the top-level weights, and (3) does not require fixing any parameters—both the input and parameter embeddings can be learned.

• NTK regime limitation: The Neural Tangent Kernel (Jacot et al., 2020) provides a kernel description of infinitely wide neural networks during training, but this linearization is accurate only when weights remain close to their initialization—the "lazy training" regime. The paper explicitly distinguishes itself: "NTK-based analysis, as leveraging Taylor-based linearization of the NN, is valid only for the mature stage of training/finetuning when weights do not change much... SNNKs do not rely on this assumption." SNNKs can be trained from scratch with weights moving arbitrarily far from initialization.

• No systematic compression mechanism: Even when kernels are connected to neural networks in prior work, the connection does not yield a compression recipe. The kernel formulation typically adds computational overhead (you need to compute kernel evaluations between all data points) rather than reducing it. The paper's bundling process (Section 3.3) provides the missing mechanism: by iteratively applying the two-tower factorization, layers can be collapsed into a single compressed representation.

3. Parameter-Efficient Fine-Tuning: Ad-Hoc Bottlenecks Without Kernel Motivation

Adapter layers (Houlsby et al., 2019) and related methods like LoRA (Hu et al., 2022) achieve parameter efficiency by introducing low-rank bottlenecks—essentially factorizing a weight matrix update $\Delta W = AB$ where $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times d}$ with $r \ll d$. This works well in practice, but:

• The bottleneck dimension is chosen heuristically: The rank $r$ is a hyperparameter selected by trial and error. There is no principled connection to the function being approximated.

• The factorization is purely linear: LoRA and adapters insert linear bottlenecks. The paper's SNNK approach, in contrast, uses nonlinear random feature maps (e.g., ReLU applied to Gaussian projections, as in the ReLU-SNNK layer of Section 3.2). This means SNNKs can capture relationships that low-rank linear factorizations cannot—a point the paper makes formally in Theorem 3.2 and Remark 3.3.

• No connection to the original layer's computation: Adapters are additive modules that learn a residual correction. SNNKs, by contrast, aim to approximate the original feedforward computation itself, which means they can serve as drop-in replacements for entire layers rather than additional modules.

How This Paper Positions Itself

The paper's positioning can be understood through several key design choices that distinguish it from prior work:

From "kernel as approximation target" to "kernel as computational primitive." Prior RF work asks: "Given a kernel $K$, how do we compute $K(x, y)$ cheaply?" This paper asks: "Given a neural network layer computing $f(Wx + b)$, can we rewrite it as computing a kernel evaluation and thereby gain computational advantages?" The direction is reversed—instead of starting with a kernel and approximating it, the paper starts with a neural network computation and discovers the kernel it implicitly defines. This is a conceptual shift with practical consequences: the goal is not to approximate $K_f(x, (w, b))$ as accurately as possible, but to create a computationally efficient module that behaves like the original FFL in the context of the larger network. This allows for approximations that would be unacceptable for kernel estimation (e.g., using only $m = 16$ random features in a 768-dimensional space) but work well in practice because the network can adapt to the approximation error.

URFs as a unifying mechanism for arbitrary activations. The paper's universal random features mechanism (Section 3.1) is designed to handle any activation function $f$ with a well-defined Fourier transform, including those where the transform exists only in the distributional sense (e.g., sine and cosine, which use Dirac delta distributions). This is critical for practical neural networks, where activations like ReLU do not have standard Fourier transforms and require distributional treatment. The decomposition into four non-negative components (Equation 6) handles the general case where $f$ is neither even, odd, nor positive-valued, and the resulting kernel $K_f$ is neither symmetric nor positive semi-definite. The paper notes that "mappings $\Phi$ and $\Psi$ in principle are different (on expectation they can produce both symmetric and asymmetric functions)," which explicitly accommodates non-positive definite and asymmetric kernels (Ong et al., 2004; He et al., 2023).

Bundling as a new compression primitive. The bundling process (Section 3.3, Figure 1 bottom) is the paper's most architecturally novel contribution. It provides an iterative procedure for compactifying deep networks: replace the first FFL with its SNNK analogue, absorb the resulting parameter embedding into the next layer's weight matrix, and repeat. This is not simply applying SNNK to each layer independently—it changes the network architecture by reducing the number of layers. The extreme case, applied to all layers, yields $y = Wx$ where $x = \Phi_{f_L}(\Phi_{f_{L-1}}(...\Phi_{f_1}(x_0)...))$ and $W$ is an unstructured learnable matrix (Equation 14). This is a genuinely new capability: for regression tasks with mean squared error loss, the optimal $W$ has a closed-form solution, "opening a possibility to bypass backpropagation" entirely. The paper demonstrates this concretely in Appendix J.3, where a bundled BERT pooler+classifier achieves competitive GLUE performance while training only ~2K parameters with a closed-form solution on the STSB regression task.

Empirically bridging kernel methods and Transformers at scale. The paper's experiments span an unusually wide range: microscopic kernel estimation (pointwise, $d = 2000$, verifying that $m \ll d$ random features suffice for accurate approximation), small-scale toy tasks (SIREN image fitting, QPNN physics simulation, MNIST classification), and large-scale Transformer fine-tuning (GLUE with BERT-base, CIFAR and ImageNet with ViT). This breadth is deliberate—it demonstrates that the SNNK concept scales from theoretical kernel approximation to practical deep learning workloads. The adapter experiments (Section 4.3.2) are particularly significant because they show SNNKs working in the regime where they are most practically useful: parameter-efficient fine-tuning of frozen large models, where reducing trainable parameters directly reduces storage, communication, and optimization costs. The paper reports a ~5× parameter reduction (0.3M vs. 0.9M trainable parameters) while outperforming standard adapters on 5 of 8 GLUE tasks (Table 1).

The expressiveness claim as a theoretical differentiator. The paper makes a specific theoretical claim that goes beyond efficiency: SNNKs are strictly more expressive than regular FFLs. The ReLU-SNNK layer (Section 3.2), defined via $\Phi(x) = \text{ReLU}(\frac{1}{\sqrt{l}}Gx)$ and $\Psi(w, b) = \text{ReLU}(\frac{1}{\sqrt{l}}Gw)$ for Gaussian $G$, corresponds via Theorem 3.2 to the first-order arc-cosine kernel $K_1(x, w) = \frac{1}{\pi}\|x\|_2\|w\|_2 J_1(\alpha_{x,w})$. The paper observes in Remark 3.3: "The ReLU-SNNK layer is not a regular FFL since the values of its output dimensions cannot be rewritten as $f(x^\top w_i + b_i)$ for some $f: \mathbb{R} \to \mathbb{R}$." This means SNNKs can represent input-output relationships that no standard FFL can, even with arbitrary activation functions. This is not merely a compression technique operating within the representational capacity of FFLs—it expands that capacity while simultaneously reducing parameters.

In summary, the paper positions SNNKs as a principled synthesis of kernel methods and deep learning that addresses a specific, practically important problem (FFL computational cost) through a mechanism (two-tower disentanglement) that prior work could not realize due to restrictive assumptions about kernels, limited scope of network-kernel connections, and the absence of a compression procedure (bundling) that exploits the disentanglement for architectural efficiency.

3. Technical Approach

3.1 Reader Orientation

We are building a drop‑in replacement for a standard feed‑forward layer f(Wx + b) that keeps exactly the same function but separates the inputs x from the learnable parameters (W,b) into two independent “towers,” connecting them only at the very end through a dot‑product. The problem it solves is that regular feed‑forward layers entangle computation and parameter count—every forward pass requires O(ld) operations and stores O(ld) parameters—whereas the two‑tower design can be instantiated with a much smaller number m of random features, reducing both training parameters (from O(ld) to O(lm)) and inference cost (from O(ld) to O(lm + md)) whenever m is substantially smaller than both l and d. The “shape” of the solution is: find random feature maps Φ_f(x) and Ψ_f(w,b) such that E[Φ_f(x)^T Ψ_f(w,b)] = f(w^T x + b), then drop the expectation and parametrise the layer as SNNK_f(x,(W,b)) = Φ_f(x)^T Ψ_f(w,b); at training time you can skip Ψ_f entirely and learn the vectors Ψ_f(w_i,b_i) directly as unstructured parameters, yielding compression.

3.2 Big‑Picture Architecture (Diagram in Words)

The system has five major parts, but in a deployed network only three are visible:

1. Universal Random Feature (URF) constructor (Section 3.1) — an offline recipe that, given an activation function f with a well‑defined Fourier transform (classical or distributional), produces the embedding functions Φ_f : R^d → R^m and Ψ_f : R^d × R → R^m. This is a mathematical derivation; it is not a runtime component.

2. Input tower Φ_f — a stateless transformation that maps a d‑dimensional input vector x to an m‑dimensional random feature vector. At inference time this tower costs O(md) to compute.

3. Parameter tower Ψ_f — turns each row w_i of the weight matrix W (plus the bias b_i) into an m‑dimensional vector. In the “SNNK as FFL replacement” mode these vectors are computed once from the learned W,b and stored; in the “compressed” mode they are directly learned as unstructured m‑dimensional vectors ω_i, bypassing Ψ_f entirely and reducing the parameter count from O(ld) to O(lm).

4. SNNK layer (the dot‑product junction) — given the input embedding Φ_f(x) and the list of parameter embeddings ω_0,…,ω_{l-1}, the layer outputs the vector of dot‑products (Φ_f(x)^T ω_0, …, Φ_f(x)^T ω_{l-1}). This costs O(lm).

5. Bundling procedure (Section 3.3) — an iterative architectural compression that replaces the first feed‑forward layer of a deep stack with an SNNK, absorbs the resulting parameter‑embedding matrix into the next layer’s weight matrix, and repeats, ultimately collapsing L feed‑forward layers into a single matrix W acting on a pre‑processed input x.

Information flow in a single SNNK layer (inference):
Input x ∈ R^d → Φ_f(x) ∈ R^m (input tower) ·— · previously computed ω_i ∈ R^m for i = 0,…,l-1 (parameter tower) → (Φ_f(x)^T ω_i) for each i → l‑dimensional output.
Training flow (compressed mode): input x → Φ_f(x); the ω_i vectors are trainable parameters updated by gradient descent; back‑propagation goes through the dot‑product junction but never through Ψ_f, because Ψ_f is never applied.

3.3 Roadmap for the Deep Dive

• First, the mathematical target: the two‑tower kernel formulation — because every subsequent design choice is driven by the requirement that E[Φ_f(x)^T Ψ_f(w,b)] = f(w^T x + b).
• Second, Universal Random Features (URFs) — this is the core algorithm that constructs Φ_f and Ψ_f for an arbitrary activation f. We walk through the Fourier decomposition, the introduction of proposal distributions, the connection to the softmax kernel, and the final random feature estimator.
• Third, the ReLU‑SNNK layer — because it breaks the mould: it is not derived from a known feed‑forward layer but instead instantiates an arc‑cosine kernel, demonstrating that SNNKs are strictly more expressive than regular FFLs.
• Fourth, the bundling process — the architectural compression that iteratively applies SNNKs to reduce the depth and parameter count of a network.
• Fifth, the time‑complexity and parameter‑compression arithmetic — with concrete numbers that show why any of this matters: how m ≪ min(l,d) translates into wall‑clock savings.

3.4 Detailed, Sentence‑Based Technical Breakdown

This is primarily a methods paper whose core idea is that a feed‑forward layer f(Wx + b) can be reinterpreted as computing the expected value of a dot‑product kernel between separate embeddings of the input and the parameters, and that this reinterpretation yields both parameter compression and computational savings when the expectation is replaced by a Monte‑Carlo estimate with a small number m of random samples.

---

The Two‑Tower Kernel Formulation: What We Are Building Toward

The paper begins by rewriting the scalar output of a single neuron as a kernel evaluation. For a single weight vector w ∈ R^d and bias b ∈ R, define:

$K_f(x, (w, b)) = \mathbb{E}\left[\Phi_f(x)^\top \Psi_f(w, b)\right]$

where Φ_f : R^d → R^m is the input embedding map, Ψ_f : R^d × R → R^m is the parameter embedding map, and the expectation is taken over the randomness used to construct these maps (the random projections). The requirement is that the expectation exactly equals the activation output:

$f(w^\top x + b) = \mathbb{E}\left[\Phi_f(x)^\top \Psi_f(w, b)\right]$

What this computes: The left‑hand side is the standard feed‑forward computation for one neuron—input dot‑product with weights plus bias, passed through activation f. The right‑hand side re‑expresses this as the expected dot‑product of two random feature vectors, one derived purely from the input x and one derived purely from the parameters (w,b).

Why this form: If this equality holds (in expectation), then we can instantiate the full layer as:

$\text{SNNK}_f(x, (W, b)) = \left(\Phi_f(x)^\top \Psi_f(w_0, b_0), \ldots, \Phi_f(x)^\top \Psi_f(w_{l-1}, b_{l-1})\right)^\top$

and this will be an unbiased estimator of the original FFL output f(Wx + b). The immediate gains are:

• Disentanglement: the computation is split into two independent stages—embed the input, embed the parameters, then dot‑product.
• Compressibility: since Ψ_f(w_i, b_i) produces an m‑dimensional vector for each neuron, and m can be chosen independently of d, we can set m ≪ d and directly learn these m‑dimensional vectors rather than the original d‑dimensional weight vectors plus bias.
• Sub‑quadratic time: if Φ_f(x) can be computed in O(md) time and m ≪ min(l,d), the total cost O(md + lm) is smaller than the original O(ld).

The critical property that makes this workable is that K_f does not need to be positive semi‑definite. Unlike classical kernel methods where the kernel must define a valid RKHS, here f can be any activation function (ReLU, tanh, sigmoid, sine, cosine) because the mappings Φ_f and Ψ_f are allowed to be different—they can produce asymmetric kernels on expectation. This is stated explicitly: "mappings Φ and Ψ in principle are different (on expectation they can produce both symmetric and asymmetric functions)" (Section 2, end).

---

Universal Random Features (URFs): Constructing Φ_f and Ψ_f for Arbitrary Activations

Step 1: Fourier Decomposition into Four Non‑Negative Parts

The construction starts from the Fourier inversion formula. For a function f : R → R whose Fourier transform FT_f exists (in the classical Riemannian sense, or in the distributional sense for functions like sine/cosine):

$f(z) = \int_{\mathbb{R}} FT_f(\xi) \exp(2\pi i \xi z) \, d\xi$

where FT_f(ξ) = ∫_R f(z) exp(-2π i ξ z) dz is the Fourier transform.

What this computes: This is the standard inverse Fourier transform—it reconstructs the original function f at point z by integrating over all frequencies ξ, with each frequency component exp(2π i ξ z) weighted by the Fourier coefficient FT_f(ξ).

Why this form: The exponential exp(2π i ξ z) will become exp(2π i ξ (w^T x + b)) when we substitute z = w^T x + b. This is crucial because the term exp(2π i ξ w^T x) factorises into a function of x times a function of w—which is exactly what disentanglement requires.

Since FT_f is in general complex‑valued and neither purely real nor non‑negative, the paper decomposes it into four non‑negative real‑valued components:

$FT_f = FT_f^{\text{re},+} - FT_f^{\text{re},-} + i \, FT_f^{\text{im},+} - i \, FT_f^{\text{im},-}$

where each of the four functions maps R → R_{≥0} (is everywhere non‑negative). The subscripts denote: re,+ = positive part of the real component, re,- = negative part of the real component, im,+ = positive part of the imaginary component, im,- = negative part of the imaginary component. Without loss of generality, the paper assumes all four are not identically zero (if any are, they are simply dropped from the sum).

What this achieves: By splitting the complex Fourier transform into four non‑negative components, we can treat each component as an unnormalised probability density (up to normalisation). This allows us to rewrite the integral as a mixture of expectations over proper probability distributions.

Step 2: Introducing Proposal Distributions and Importance Sampling

For each component j ∈ {0,1,2,3}, define a proposal distribution \bar{P}_j on R with density \bar{p}_j (e.g., Gaussian). Also define the target distribution P_j with density p_j proportional to the corresponding Fourier component:

$p_j(\xi) \propto FT_f^{(\cdot)}(\xi)$

where FT_f^{(\cdot)} is the relevant non‑negative component. The normalising constants are:

$c_0 = \int_{\mathbb{R}} FT_f^{\text{re},+}(\tau) d\tau, \quad c_1 = -\int_{\mathbb{R}} FT_f^{\text{re},-}(\tau) d\tau, \quad c_2 = i\int_{\mathbb{R}} FT_f^{\text{im},+}(\tau) d\tau, \quad c_3 = -i\int_{\mathbb{R}} FT_f^{\text{im},-}(\tau) d\tau$

What these normalising constants are: Each c_j is the total integral (or its negative, or multiplied by i or -i) of the corresponding non‑negative component. They ensure that when we rewrite the Fourier integral as an expectation, the scaling is correct.

Using importance sampling (multiply and divide by \bar{p}_j(ξ)), the reconstruction becomes:

$f(z) = \sum_{j=0}^3 c_j \, \mathbb{E}_{\xi \sim \bar{P}_j}\left[\frac{p_j(\xi)}{\bar{p}_j(\xi)} \exp(2\pi i \xi z)\right]$

What this computes: This rewrites the integral ∫ FT_f(ξ) exp(2π i ξ z) dξ as a sum of four expectations, each over its own proposal distribution \bar{P}_j. The importance weight p_j(ξ)/\bar{p}_j(ξ) corrects for the fact that we are sampling from \bar{P}_j rather than the target P_j. The complex exponential exp(2π i ξ z) is the signal we want to disentangle.

Why importance sampling is necessary: We cannot sample directly from the target distribution P_j because its density p_j is proportional to a Fourier component that may be difficult to normalise or sample from. The proposal distribution \bar{P}_j can be any easy‑to‑sample distribution (e.g., Gaussian), and the importance weight corrects for the mismatch. The paper notes that in the special case of discrete P_j with a finite number of atoms (e.g., for sine/cosine activations, see below), one can sample directly without importance weights—each atom is used deterministically with its probability mass.

Step 3: Substituting z = w^T x + b and Identifying the Softmax Kernel

Now substitute z = w^T x + b:

$f(w^\top x + b) = \sum_{j=0}^3 c_j \, \mathbb{E}_{\xi \sim \bar{P}_j}\left[\frac{p_j(\xi)}{\bar{p}_j(\xi)} \exp(2\pi i \xi (w^\top x + b))\right]$

Factor the exponential as:

$\exp(2\pi i \xi (w^\top x + b)) = \exp(2\pi i \xi b) \cdot \exp\left((2\pi i \xi x)^\top w\right)$

To symmetrise the roles of x and w, introduce two complex scaling factors ρ(ξ) and η(ξ) such that ρ(ξ) η(ξ) = 2π i ξ. The paper states they choose ρ(ξ) = 2π i ξ and η(ξ) = 1 in experiments, but the formulation is general. Define:

$\hat{x}(\xi) = \rho(\xi) x, \quad \hat{w}(\xi) = \eta(\xi) w$

Then the exponential becomes:

$\exp(2\pi i \xi (w^\top x + b)) = \exp(2\pi i \xi b) \cdot \exp\left(\hat{x}(\xi)^\top \hat{w}(\xi)\right)$

What just happened: We have rewritten exp(2π i ξ (w^T x + b)) as (scalar factor depending on b) × exp(ŷ_x(ξ)^T ŷ_w(ξ)). The inner exp(ŷ_x^T ŷ_w) is exactly the softmax kernel K_exp(ŷ_x, ŷ_w) = exp(ŷ_x^T ŷ_w) evaluated at the transformed vectors ŷ_x(ξ) and ŷ_w(ξ). This is the critical bridge: we have reduced the problem of constructing random features for an arbitrary activation f to the problem of constructing random features for the softmax kernel.

Let S_j(ξ, b) = (p_j(ξ)/\bar{p}_j(ξ)) exp(2π i ξ b). Then:

$f(w^\top x + b) = \sum_{j=0}^3 c_j \, \mathbb{E}_{\xi \sim \bar{P}_j}\left[S_j(\xi, b) \cdot K_{\text{exp}}(\hat{x}(\xi), \hat{w}(\xi))\right]$

Step 4: Linearising the Softmax Kernel

The paper leverages the softmax‑kernel linearisation from Likhosherstov et al. (2022):

$\exp(\hat{x}^\top \hat{w}) = \mathbb{E}_{g \sim \mathcal{N}(0, I_d)}\left[\Lambda_g(\hat{x}) \, \Lambda_g(\hat{w})\right]$

where, for any hyperparameter A ≤ 0, the function Λ_g : R^d → R is defined as:

$\Lambda_g(z) = (1 - 4A)^{d/4} \exp\left(A\|g\|_2^2 + \sqrt{1 - 4A} \, g^\top z - \frac{\|z\|_2^2}{2}\right)$

What Λ_g(z) does: Given a random Gaussian vector g ∼ N(0, I_d), this function maps any input z ∈ R^d to a scalar such that the expected product E_g[Λ_g(ŷ_x) Λ_g(ŷ_w)] equals exp(ŷ_x^T ŷ_w). It is a positive random feature map for the softmax kernel—meaning the features are real‑valued (not complex) and the estimator is unbiased.

What the hyperparameter A does: A ≤ 0 controls a variance‑bias tradeoff in the approximation. Setting A < 0 makes the features bounded (Remark 3.1), which is important for the error propagation analysis (Appendix B) because bounded features enable Azuma’s inequality concentration guarantees. The paper uses A < 0 in practice.

Why this particular map: Likhosherstov et al. (2022) proposed this as an improvement over earlier softmax random features (Choromanski et al., 2021) that offers better variance properties and boundedness guarantees. The boundedness is critical for neural network applications where inputs and weights can be large—unbounded features would produce unstable training and unreliable estimates.

Step 5: Assembling the Final Random Feature Vectors

Putting everything together, for each component j:

$f_j(x, w, b) = c_j \, \mathbb{E}_{\xi \sim \bar{P}_j}\left[S_j(\xi, b) \cdot \mathbb{E}_{g \sim \mathcal{N}(0,I_d)}\left[\Lambda_g(\hat{x}(\xi)) \, \Lambda_g(\hat{w}(\xi))\right]\right]$

By Fubini’s theorem, the two expectations can be combined into a single expectation over the product distribution \bar{P}_j ⊗ N(0, I_d):

$f_j(x, w, b) = \mathbb{E}_{(\xi, g) \sim \bar{P}_j \otimes \mathcal{N}(0, I_d)}\left[\Gamma^1_{g,\xi}(x) \cdot \Gamma^2_{g,\xi}(w, b)\right]$

where the per‑sample scalar features are:

$\Gamma^1_{g,\xi}(x) = \Lambda_g(\rho(\xi) x)$ $\Gamma^2_{g,\xi}(w, b) = c_j \, S_j(\xi, b) \, \Lambda_g(\eta(\xi) w)$

What these are: Γ^1_{g,ξ}(x) is a scalar computed from the input x using the random Gaussian vector g and the frequency sample ξ. Γ^2_{g,ξ}(w,b) is a scalar computed from the weights w and bias b using the same g and ξ, plus the importance weight S_j and the normalising constant c_j. Their product is an unbiased estimator of the j‑th component’s contribution to f(w^T x + b).

Why this factorisation works: The input x appears only in Γ^1 and the parameters (w,b) appear only in Γ^2. The random variables (ξ, g) are shared. This is the exact disentanglement the paper aims for.

Step 6: Monte Carlo Instantiation with m Samples

Draw m independent samples (ξ_1, g_1), …, (ξ_m, g_m) ∼ \bar{P}_j ⊗ N(0, I_d). Define the m‑dimensional vectors:

$\Phi_j(x) = \frac{1}{\sqrt{m}}\left(\Gamma^1_{g_1,\xi_1}(x), \ldots, \Gamma^1_{g_m,\xi_m}(x)\right)^\top$ $\Psi_j(w, b) = \frac{1}{\sqrt{m}}\left(\Gamma^2_{g_1,\xi_1}(w, b), \ldots, \Gamma^2_{g_m,\xi_m}(w, b)\right)^\top$

Then Φ_j(x)^T Ψ_j(w,b) = (1/m) Σ_{k=1}^m Γ^1_{g_k,ξ_k}(x) Γ^2_{g_k,ξ_k}(w,b) is an unbiased estimator of f_j(x,w,b) with variance scaling as 1/m.

The 1/√m normalisation: Each entry of Φ_j and Ψ_j contains 1/√m so that the dot‑product yields 1/m × sum, matching the Monte Carlo average.

Finally, concatenate the four components (j = 0,1,2,3) to get the full feature maps:

$\Phi_f(x) = \text{concat}(\Phi_0(x), \Phi_1(x), \Phi_2(x), \Phi_3(x))$ $\Psi_f(w, b) = \text{concat}(\Psi_0(w, b), \Psi_1(w, b), \Psi_2(w, b), \Psi_3(w, b))$

What this produces: The final embedding vectors are of dimension 4m (or less if some components are identically zero—e.g., for odd/even functions, two of the four components vanish). The dot‑product Φ_f(x)^T Ψ_f(w,b) is an unbiased estimator of f(w^T x + b). For the full layer with l neurons, each neuron i gets its own Ψ_f(w_i, b_i) vector, and the output is the l‑vector of dot‑products.

Sampling strategies: The paper mentions two variants:

• IID sampling: each (ξ_k, g_k) drawn independently.
• Block‑IID sampling: use a fixed ξ within a block of samples but draw g independently; different blocks use independent ξ. This can reduce the number of Fourier samples needed while maintaining diversity through the Gaussian samples.

Special Case: Trigonometric Activations (Sine and Cosine)

When f(z) = sin(z) or f(z) = cos(z), the Fourier transform exists only in the distributional sense as Dirac delta distributions:

$\sin(z) = \frac{e^{iz} - e^{-iz}}{2i}, \quad \cos(z) = \frac{e^{iz} + e^{-iz}}{2}$

This means the corresponding probability distributions P_j are discrete binary distributions over the two atoms {-1/(2π), +1/(2π)}. No importance sampling is needed—one can deterministically use these two frequencies. The paper notes: "this observation has interesting practical consequences, since it leads to the conceptually simple linearization of the FFLs applied in SIREN networks." Concretely, for sine/cosine SNNK layers (used in the SIREN and QPNN experiments in Section 4.2), the URF construction simplifies considerably: the proposal distributions collapse to two atoms, and the random feature maps are built only from Gaussian samples g with no Fourier sampling variance.

Boundedness Guarantee (Remark 3.1)

When A < 0 in the Λ_g definition, and when ∥x∥_2, ∥w∥_2, and |b| are upper‑bounded, all entries of Φ_f(x) and Ψ_f(w,b) are also upper‑bounded. This follows directly from the form of Λ_g(z) in Equation 8: the term exp(A ∥g∥_2^2) with A < 0 ensures exponential decay as ∥g∥_2^2 grows, and the remaining terms are bounded when z is bounded. This boundedness is essential for concentration guarantees (Appendix B, Equation 18) and for stable training—it prevents the random feature values from exploding, which would cause gradient variance to spike.

Why this matters for deployment: In a neural network, inputs x and weights w may be unbounded in principle, but in practice they are finite. The positive random feature mechanism with A < 0 provides a guarantee that the feature values stay within a known range, which enables the Azuma inequality bound in Equation 18 that controls the probability of large approximation errors across the bundled network.

---

The ReLU‑SNNK Layer: Beyond Regular Feed‑Forward Layers

The paper introduces a second SNNK variant that is not derived from linearising a standard f(w^T x + b) FFL. Instead, it instantiates an arc‑cosine kernel.

Definition (Section 3.2): For a Gaussian matrix G ∈ R^{l×d} with entries sampled independently from N(0,1), define:

$\Phi(x) = \text{ReLU}\left(\frac{1}{\sqrt{l}} G x\right), \quad \Psi(w, b) = \text{ReLU}\left(\frac{1}{\sqrt{l}} G w\right)$

The bias b is ignored in this construction. The output of the ReLU‑SNNK layer for the i‑th neuron is Φ(x)^T Ψ(w_i, b_i).

What this computes: The input x is projected by the Gaussian matrix G, scaled by 1/√l, and passed through ReLU to produce an l‑dimensional feature vector. The same Gaussian matrix is applied to the weight vector w_i (but not to the input), and the ReLU is applied. The dot‑product of these two l‑dimensional vectors is the neuron output. The Gaussian matrix G is fixed—it is not learned.

Connection to arc‑cosine kernels (Theorem 3.2): Theorem 3.2 (proved in Cho & Saul, 2011) states that the n‑th order arc‑cosine kernel K_n(x, y) = (1/π) ∥x∥_2^n ∥y∥_2^n J_n(α_{x,y}), where α_{x,y} is the angle between x and y, can be linearised as:

$K_n(x, y) = 2 \, \mathbb{E}_{ω \sim \mathcal{N}(0, I_d)}\left[\Gamma_n(x)^\top \Gamma_n(y)\right]$

where Γ_n(v) = \text{ReLU}((v^\top ω)^n). For n = 1, Γ_1(v) = \text{ReLU}(v^\top ω). The expectation over ω ∼ N(0, I_d) is approximated by averaging over the l rows of the Gaussian matrix G (each row is an independent Gaussian vector). The scaling 1/√l in the definition makes the dot‑product Φ(x)^T Ψ(w_i) an unbiased estimator of (1/2) K_1(x, w_i).

What this means: The ReLU‑SNNK layer computes:

$\text{ReLU-SNNK}(x, (W, b))_i = \frac{1}{2} K_1(w_i, x) \quad \text{(in expectation)}$

where K_1 is the first‑order arc‑cosine kernel. The output depends on the angle between x and w_i through the function J_1(α) = sin(α) + (π - α) cos(α), multiplied by the product of norms ∥x∥_2 ∥w_i∥_2.

Why this is not a regular FFL (Remark 3.3): The paper explicitly claims: "The ReLU‑SNNK layer is not a regular FFL since the values of its output dimensions cannot be rewritten as f(x^T w_i + b_i) for some f : R → R." The key observation is that the arc‑cosine kernel depends on the angle between x and w_i and on the norms ∥x∥_2 and ∥w_i∥_2. This is a more complex function than any pointwise transformation of the dot‑product x^T w_i. Dot‑product alone determines the cosine of the angle, but not the individual norms—two pairs of vectors (x, w) and (x', w') can have the same dot‑product but different norms, and thus different arc‑cosine kernel values.

Why this expressiveness matters: The ReLU‑SNNK layer demonstrates that the SNNK framework is not merely a compression technique—it can represent input‑output relationships that are fundamentally outside the representational capacity of standard FFLs. This is a theoretical strength: adopting SNNKs does not constrain you to a subset of what FFLs can do; it potentially expands what you can model. The paper uses the ReLU‑SNNK for adapter experiments (Section 4.3.2) where it proves particularly effective—outperforming standard adapters on 5 of 8 GLUE tasks (Table 1).

---

Neural Network Bundling: Iterative Architectural Compression

The bundling process (Section 3.3) is the mechanism that exploits the two‑tower formulation to compactify multiple feed‑forward layers, not just one. The key insight is that once the first layer is rewritten in SNNK form, its parameter‑embedding matrix can be absorbed into the weight matrix of the next layer.

Step 1: Replace the first FFL with its SNNK analogue. Starting from an L‑layer network:

$x_{i+1} = f_{i+1}(W_i x_i + b_i), \quad i = 0, \ldots, L-1$

with x_0 = x (the network input), replace the first layer (i = 0) with its SNNK version. The first layer’s output is:

$\widehat{x}_1 = \Phi_{f_1}(x_0)^\top \Psi_{f_1}(W_0, b_0)$

where Ψ_{f_1}(W_0, b_0) is now a matrix whose columns are the embedding vectors Ψ_{f_1}(w_0^j, b_0^j) for each neuron j = 0, ..., d_1-1 (here w_0^j is the j‑th row of W_0). This matrix has dimensions m × d_1.

Step 2: Absorb into the next layer. The output ŷ_x_1 is m‑dimensional (since Φ_{f_1}(x_0) is m‑dimensional and Ψ_{f_1}(W_0, b_0) is m × d_1, the product is d_1‑dimensional—matching the original first layer’s output). The second layer originally takes x_1 ∈ R^{d_1} as input and applies W_1 ∈ R^{d_2 × d_1}. In the bundled version, define:

$\widehat{W}_0 = W_1 \Psi_{f_1}(W_0, b_0) \in \mathbb{R}^{d_2 \times m}$

and set ŷ_x_0 = Φ_{f_1}(x_0) ∈ R^m. Then Ŵ_0 ŷ_x_0 is the input to the activation f_2 of the second layer.

What happened: We have replaced a module of two feed‑forward layers with a module that has one feed‑forward layer (the second original layer, slightly modified) preceded by a feature map Φ_{f_1}. The first layer’s weight matrix W_0 has been absorbed into a single matrix Ŵ_0 that combines it with W_1.

Step 3: Iterate. The new block ŷ_x_{i+1} = ŷ_{i+1}(Ŵ_i ŷ_x_i + ŷ_i) has the same form as the original network but with one fewer layer. Applying the same procedure iteratively to the first layer of the remaining stack yields, after L steps:

$y = W x$

where:

$x = \Phi_{f_L}\left(\Phi_{f_{L-1}}\left(\ldots \Phi_{f_1}(x_0) \ldots\right)\right) \in \mathbb{R}^m$ $W = \Psi_{f_L}\left(W_{L-1} \Psi_{f_{L-1}}\left(\ldots W_2 \Psi_{f_2}\left(W_1 \Psi_{f_1}(W_0, b_0), b_1\right) \ldots\right), b_{L-1}\right) \in \mathbb{R}^{d_L \times m}$

What this produces: The entire deep network has been collapsed into a linear model y = W x where the input x is a highly non‑linear feature representation of the original input (composed of all the Φ maps) and W is an unstructured matrix. At training time, x is fixed (since the Φ maps use fixed random projections) and only W is learned. The number of trainable parameters is d_L × m instead of the original Σ_i d_{i+1} × d_i.

Explicit solution for the optimal W: Since y = W x is a linear model, for several loss functions the optimal W can be written in closed form. In particular, for mean squared error regression with training data {x^{(k)}, y^{(k)}}:

$W_{\text{opt}} = Y X^\top (X X^\top)^{-1}$

where X has columns x^{(k)} and Y has columns y^{(k)}. The paper demonstrates this concretely on the STSB regression task (Appendix J.3): "we have a closed form formula for W in that case. We computed the features of the BERT vectors via our random feature mechanism and calculated the coefficients of the linear regressor using the closed form formula, which gives us a Spearman correlation of 67. The entire fitting and evaluation takes barely a few seconds on a CPU."

Selective bundling: The paper emphasises that bundling can be applied to a subset of layers (not necessarily all), targeting a specific module for compression while leaving other layers intact. This is what the uptraining experiments do (Appendix J.4): only the top k FFN blocks of the Transformer are replaced with bundled SNNKs, compressing those blocks while keeping the rest of the network unchanged.

Error propagation (Appendix B): When bundling multiple layers, the variance from each layer’s random feature approximation accumulates. The paper analyses this: if the kernel function K is Lipschitz‑continuous in the region of interest, and the number of random features per layer is m, then the probability that the bundled d‑layer network’s output differs from the exact output by more than ε + δ(ε) + … (composition of d-1 Lipschitz bounds) is at most d times the single‑layer error probability from the Azuma bound (Equation 18). The implication: deeper bundling requires either more random features per layer or more careful control of Lipschitz constants to maintain accuracy.

---

Time Complexity and Parameter Compression: The Arithmetic That Makes It Work

Time complexity of a single SNNK layer (given pre‑computed parameter embeddings):

Let t_m(d) be the time to compute Φ_f(x) for an input x ∈ R^d. With the URF construction, t_m(d) = O(md) because computing each of the m entries of Φ_f(x) requires evaluating Λ_g(ρ(ξ)x), which involves a dot‑product g^T x (costing O(d)) plus scalar operations. With m such entries, the total is O(md).

Given pre‑computed vectors Ψ_f(w_i, b_i) for i = 0, …, l-1, the dot‑product Φ_f(x)^T Ψ_f(w_i, b_i) costs O(m) per neuron, for a total of O(lm) for the full layer.

Thus the total inference time is:

$T_{m,l}(d) = O(md + lm) = O(m(d + l))$

Compare this to the original FFL cost O(ld). The SNNK is faster when m(d + l) < ld, i.e., when:

$m < \frac{ld}{d + l} = \frac{1}{1/l + 1/d}$

In the regime where l ≪ d or d ≪ l, the threshold is m < min(l, d). For the typical case where l and d are comparable (e.g., in Transformer FFNs where the hidden dimension is 4d), the threshold is m < l/2 (roughly). The paper uses m = 8 or 16 for d = 768 (BERT hidden size), which satisfies m ≪ d and m ≪ l (since l ≈ 3072 for the FFN intermediate layer), yielding clear computational savings.

Parameter compression: In the compressed training mode, the Ψ_f map is never applied. Instead, for each of the l neurons, we directly learn an m‑dimensional vector ω_i. The total number of trainable parameters for the layer is O(lm) instead of O(ld). The compression ratio is approximately d/m. For m = 16 and d = 768, this is roughly a 48× reduction in parameters per layer.

For the adapter experiments (Section 4.3.2), the paper uses m = 16 random features, resulting in (d + 1) × m trainable parameters when including the gating vector. For BERT‑base with d = 768, this is 769 × 16 ≈ 12,304 parameters per adapter module, compared to the standard Houlsby adapter with bottleneck dimension 48, which uses 2 × 48 × 768 ≈ 73,728 parameters—about a 6× reduction. Across all layers, the paper reports roughly a 3× reduction in total trainable parameters (Table 1, Figure 4).

The gating vector v: In the adapter implementation (Appendix D), the SNNK output is modulated by a learned vector v ∈ R^d initialised to zero:

$Y = (v \odot \text{SNNK}(X)) + X$

where ⊙ is element‑wise multiplication. At initialisation (v = 0), the adapter block acts as the identity function, which is known to be important for stable adapter training (Houlsby et al., 2019). The gating vector adds d trainable parameters per adapter block—negligible compared to the l × m parameters of the SNNK itself.

Bundling compression: After bundling k layers, the compressed weight matrix W has dimensions d_out × m, where d_out is the output dimension of the last bundled layer and m is the number of random features. The original k layers had Σ_i d_{i+1} × d_i parameters. For the up‑training experiments (Appendix J.4), bundling 6 FFN blocks reduces the BERT model size from 440 MB to 226.71 MB (about 48% reduction) and ViT from 346 MB to 176.42 MB (about 49% reduction), while inference FLOPs drop by roughly 32–33% (Table 6).

---

Design Choices and Their Justifications: A Summary

• Fourier decomposition into four non‑negative components rather than attempting to sample from a complex distribution: Complex‑valued probability distributions are not standard. By splitting into four non‑negative components and handling each with importance sampling, the URF mechanism works with standard real‑valued proposal distributions (e.g., Gaussian) and properly handles the sign and phase of the Fourier coefficients.

• Softmax kernel linearisation (Λ_g) from Likhosherstov et al. (2022) rather than earlier methods: This choice provides bounded features when A < 0, which is essential for error propagation guarantees (Azuma bound in Appendix B) and for training stability. Earlier softmax random features (Choromanski et al., 2021) do not guarantee boundedness.

• ρ(ξ) = 2π i ξ and η(ξ) = 1: This assignment places all the frequency‑dependent scaling on the input tower and leaves the parameter tower with simple Gaussian projections. The paper notes this is an experimental choice, not a theoretical requirement—the only constraint is ρ(ξ) η(ξ) = 2π i ξ. Alternative assignments could balance the variance between the two towers differently.

• Block‑IID sampling as an alternative to full IID: Using the same ξ across a block of Gaussian samples g can reduce the number of distinct Fourier frequencies that need to be sampled while maintaining diversity through the Gaussian vectors. This is a variance‑reduction technique that the paper mentions but does not extensively ablate.

• Direct learning of ω_i instead of computing Ψ_f(w_i, b_i): This is the key practical insight. Since the Ψ_f map is just a random transformation, and the goal is to have m‑dimensional embeddings that work well in the dot‑product with Φ_f(x), one can simply learn those embeddings directly via gradient descent. This turns the SNNK from "a way to approximate an existing FFL" into "a new, more parameter‑efficient layer type that can be trained end‑to‑end."

• Gating vector initialised to zero: For adapter training, initialising the block as identity is critical for stability—the pretrained model’s behaviour is preserved at the start of fine‑tuning. The gating vector achieves this without constraining the SNNK weight matrix A to be zero (which would cause optimisation difficulties).

• Reuse of the same Gaussian matrix G for all neurons in ReLU‑SNNK: Each row g_i of G produces one entry of both Φ(x) and Ψ(w_j). The dot‑product Φ(x)^T Ψ(w_j) sums over these l entries. This is a structured sharing of randomness that makes the layer computation efficient—only one Gaussian projection matrix is stored and applied.

4. Key Insights and Innovations

Innovation 1: Reframing Feedforward Layers as Kernel Estimators—With the Express Purpose of Disentanglement and Compression, Not Kernel Approximation

The paper's most fundamental conceptual move is not a new algorithm, but a reinterpretation of what a feedforward layer is. The field already knew that two-layer networks with fixed random first-layer weights could be viewed as kernel machines (Cho & Saul, 2009; 2011; Neal, 1996), and that random features could approximate known kernels efficiently (Rahimi & Recht, 2007). But these connections ran in the "kernel-first" direction: start with a kernel, find random features to approximate it, or start with a shallow network and interpret it post-hoc as a kernel machine.

This paper reverses the arrow. It asks: given an arbitrary feedforward layer f(Wx + b), can we rewrite it as the evaluation of a kernel K_f between separate embeddings of the input and the parameters? If so, the FFL is no longer a fixed computational primitive—it becomes a family of possible instantiations, all approximating the same function, with different computational profiles. The kernel formulation is not the output of the method; it is the lens through which the layer is redesigned.

This reframing is intellectually distinctive because it changes the objective. Prior random feature work asks: "How do we approximate kernel K with low variance?" This paper asks: "How do we make this layer computationally efficient while preserving its function in the context of the larger network?" The difference matters. A kernel approximation that is unacceptably noisy for a stand-alone kernel machine (e.g., using only m = 16 random features in a 768-dimensional space) can work perfectly well when the rest of the network can compensate for the approximation error. The evidence is striking: adapter experiments use m = 16 random features and outperform standard adapters on 5 of 8 GLUE tasks (Table 1), while the MNIST SNNK-MLP with varying m achieves competitive or better accuracy than the baseline (Figure 9, left). These numbers do not make sense if SNNKs are judged purely as kernel approximators—they only make sense because the network learns around the approximation.

The significance of this reframing extends beyond the paper's empirical results. It opens a design space where any feedforward layer can be reparameterized in kernel form, and the choice of m becomes a tunable efficiency-accuracy tradeoff knob—analogous to how the rank r in LoRA (Hu et al., 2022) controls the compression-expressiveness balance, but with a principled connection to the original layer's computation rather than a heuristic low-rank assumption. The paper's explicit statement that K_f does not need to be positive semi-definite—and that Φ_f and Ψ_f can be different maps producing asymmetric kernels—is not a minor technical detail. It is the conceptual key that unlocks the full space of activation functions (ReLU, tanh, sigmoid, sine, cosine) for kernel-based reparameterization, where prior RF methods were largely restricted to positive definite, symmetric kernels.

This is a fundamental shift in perspective, not an incremental algorithm. It takes a well-known mathematical equivalence (inverse Fourier transform) and a known random feature mechanism (softmax kernel linearization from Likhosherstov et al., 2022) and combines them into a framework that redefines what a neural network layer can be: a kernel instantiated by random features, with controllable computational cost.

Innovation 2: The Universal Random Feature (URF) Mechanism as a Unifying Construction for Arbitrary Activations

Prior to this work, random feature maps for kernel approximation were developed on a case-by-case basis. Gaussian kernels used trigonometric features (Rahimi & Recht, 2007), softmax kernels used positive random features (Choromanski et al., 2021; Likhosherstov et al., 2022), polynomial kernels used tensor sketching (Kar & Karnick, 2012; Wacker et al., 2022a;b), and arc-cosine kernels used ReLU-activated Gaussian projections (Cho & Saul, 2011). Each kernel required its own derivation, its own sampling scheme, and its own variance analysis. There was no systematic procedure that, given an arbitrary activation function f, would produce the corresponding random feature maps Φ_f and Ψ_f.

The URF mechanism fills this gap. It provides a single, algorithmic template—Fourier decompose into four non-negative components, introduce proposal distributions for importance sampling, substitute z = w^T x + b, identify the softmax kernel, linearize it with Λ_g—that works for any activation function with a well-defined Fourier transform (classical or distributional). This is not merely a convenience; it is a theoretical unification. The paper demonstrates it explicitly for trigonometric activations (sine and cosine, where the Fourier transform involves Dirac deltas and the construction simplifies to a discrete two-atom distribution) and implicitly for ReLU via the arc-cosine connection (Section 3.2, Theorem 3.2). The brute-force attempt to handle tanh via polynomial kernel methods (Appendix L.1) fails—"applying a sparse kernel distorts [the pretrained features]"—underscoring that the URF approach succeeds where simpler methods do not.

What makes this distinctive is the generality-via-decomposition strategy. Rather than trying to sample from the complex-valued Fourier transform directly (which would require complex probability distributions), the URF mechanism decomposes it into four non-negative real-valued components, each of which defines a proper probability distribution after normalization. The importance sampling trick with proposal distributions \bar{P}_j means these target distributions never need to be sampled from directly—any easy-to-sample distribution (e.g., Gaussian) can be used, with the importance weight p_j(\xi) / \bar{p}_j(\xi) correcting for the mismatch. This is a standard technique in Monte Carlo methods, but applying it in the context of random feature construction for arbitrary neural network activations—and connecting it to softmax kernel linearization as the final disentanglement step—is novel.

The significance of this unification is both practical and theoretical. Practically, it means that as new activation functions are proposed for neural networks (e.g., Swish, GELU, Mish), the corresponding SNNK variant can be derived by mechanically applying the URF template, without requiring new mathematical insight for each function. Theoretically, it establishes that the class of activation functions amenable to kernel-based disentanglement is precisely the class of functions with well-defined Fourier transforms—a clean characterization that includes essentially all activations used in practice (since even ReLU can be handled via distributional Fourier transforms or the arc-cosine kernel connection).

The URF mechanism is an enabling advance—it does not directly produce accuracy gains, but it makes the entire SNNK framework possible for arbitrary activations. Without it, each new activation would require a bespoke random feature derivation, and the paper's claim of a general-purpose FFL replacement would be unsupported.

Innovation 3: The Neural Network Bundling Process as a New Compression Primitive—and the Discovery That Entire Deep Networks Can Collapse to Linear Models

The bundling process (Section 3.3) is the paper's most architecturally novel contribution, and its implications extend well beyond the empirical results reported. The core idea—iteratively replacing the first layer of a deep network with its SNNK analogue, absorbing the resulting parameter embedding into the next layer's weight matrix, and repeating—produces something genuinely unexpected: an entire deep neural network can be compressed into a single linear transformation y = Wx, where x is a fixed nonlinear feature representation of the input (composed of all the Φ maps) and W is an unstructured, learnable matrix.

This is not layer-wise compression applied independently to each layer. It is a structural transformation of the network architecture that reduces depth: an L-layer network becomes a 1-layer linear model acting on pre-processed inputs. The paper demonstrates this concretely in the extreme case (Equation 14), and validates it empirically in the bundled pooler+classifier experiments (Appendix J.3, Table 5), where a BERT-based model with a linearized pooler layer, bundled with the classifier, achieves competitive GLUE performance while training only ~2K parameters (~1/30th of the baseline).

What makes this intellectually significant is that it reveals a previously unrecognized property of feedforward networks: under the SNNK reparameterization, the nonlinearities of intermediate layers can be absorbed into the input representation rather than remaining in the computational graph. The deep network's expressiveness is not eliminated—it is relocated. The composed feature map Φ_{f_L} ∘ ... ∘ Φ_{f_1} applies fixed nonlinear transformations (random projections + activations), and the single matrix W learns to linearly combine these features. This is reminiscent of extreme learning machines (ELMs) and reservoir computing, where a fixed random nonlinear expansion is followed by a learned linear readout, but with the crucial difference that the random expansion here is derived from the original network's architecture and activations rather than chosen arbitrarily.

The closed-form solution for the optimal W under mean squared error loss is more than a curiosity. It means that for regression tasks, the bundling process can bypass backpropagation entirely: compute the fixed features x^{(k)} for all training points, then solve the linear system Y = WX using standard linear algebra. The paper's STSB demonstration (Spearman correlation of 67, computed in seconds on a CPU) is a proof-of-concept that this is not merely theoretical—it works on real tasks. This opens the door to training paradigms where deep feature extraction (via fixed random transformations) is separated from task-specific learning (via closed-form linear regression), potentially enabling extremely fast adaptation to new tasks without gradient-based optimization.

The bundling process also introduces a new axis for model compression that is orthogonal to existing methods. Pruning removes individual weights; distillation transfers knowledge to a smaller architecture; quantization reduces numerical precision. Bundling, in contrast, restructures the computational graph itself—reducing the number of layers, absorbing parameter matrices into smaller ones, and replacing learned nonlinearities with fixed random projections. The uptraining experiments (Appendix J.4, Table 6, Figures 12–13) show that bundling the top 6 FFN blocks of BERT reduces model size from 440 MB to 226.71 MB and inference FLOPs by ~32%, with minimal accuracy degradation. These savings are on top of what pruning or quantization could achieve on the already-bundled model.

This innovation is a fundamental architectural discovery—not just a compression technique, but a demonstration that deep feedforward computation can be structurally simplified in a way that was not previously recognized, with implications for training (closed-form solutions), deployment (smaller models, faster inference), and our theoretical understanding of where nonlinearities are actually needed in deep networks.

Innovation 4: SNNKs Are Strictly More Expressive Than Regular Feedforward Layers—A Theoretical Result with Practical Consequences

The paper makes a specific, provable claim that goes beyond efficiency: the ReLU-SNNK layer (Section 3.2, Theorem 3.2) computes a function of the input and weights that cannot be expressed as f(w^T x + b) for any activation function f: R → R. The proof is structural: the first-order arc-cosine kernel K_1(x, w) depends on the angle and the individual norms ∥x∥_2 and ∥w∥_2, whereas any function of the dot-product w^T x can only depend on the norms through their product ∥x∥_2 ∥w∥_2 cos(α). Two different input-weight pairs can have identical dot-products but different arc-cosine kernel values if their individual norms differ.

This is not an incremental extension of existing expressiveness results. The standard view of feedforward layers—going back to the perceptron and formalized in the universal approximation theorem literature—treats each neuron as computing f(w^T x + b). The activation function f can be anything (ReLU, sigmoid, sine, etc.), but the argument to f is always the dot-product of the input and the weight vector (plus bias). The ReLU-SNNK layer breaks this mold entirely: its computation involves a matrix of random projections applied independently to x and w, followed by a dot-product in the projected space. This is structurally more general than any f(w^T x + b) neuron, and Theorem 3.2 formalizes exactly what it computes: the arc-cosine kernel.

The significance of this result has several dimensions:

First, it refutes any concern that SNNKs are merely approximating existing FFLs with some approximation error. The ReLU-SNNK layer is not an approximation of any FFL—it is a genuinely new computational primitive that happens to be expressible within the SNNK framework (as a dot-product kernel between embedded input and parameters). The paper's phrase "strictly more expressive" is justified: the set of functions computable by SNNKs is a strict superset of those computable by standard FFLs.

Second, it explains why the ReLU-SNNK adapter performs well empirically (Table 1, bottom half: it outperforms standard adapters on 5 of 8 GLUE tasks). If SNNKs were merely compressed approximations of FFLs, one would expect them to underperform or at best match FFL-based adapters. The fact that they can outperform—while using fewer parameters—suggests that the additional expressiveness (angle-dependent, norm-sensitive computation) is genuinely useful for the downstream tasks, not just a theoretical curiosity.

Third, it opens a design space for new activation-like computations. The arc-cosine kernel is one specific instance (n = 1); Theorem 3.2 defines an entire family of order-n arc-cosine kernels K_n, each producible by a corresponding SNNK layer with Γ_n(v) = ReLU((v^T ω)^n). Higher-order kernels capture increasingly complex angular relationships. The paper does not explore n > 1 experimentally, but the theorem establishes that the SNNK framework can systematically generate layer types with expressiveness beyond standard FFLs by varying n.

Fourth, this result connects SNNKs to the literature on deep kernels and compositional kernel machines (Cho & Saul, 2009; 2011) in a new way. Prior work showed that stacking arc-cosine kernel layers mimics the hierarchical feature learning of deep networks. The SNNK formulation shows that the arc-cosine kernel can be embedded within a standard neural network architecture as a drop-in layer replacement—bridging the kernel and deep learning perspectives at the layer level rather than only at the whole-network level.

This innovation is a fundamental theoretical advance with practical implications. It establishes that the move from FFLs to SNNKs is not a tradeoff of expressiveness for efficiency, but rather an expansion of expressiveness with efficiency gains—a rare combination in machine learning methods.

Innovation 5: A Diagnostic Concept—Verifier Over-Optimization Morphs into "Approximation Error Propagation Across Bundled Layers"

While the paper does not frame it in these terms, the analysis in Appendix B constitutes a diagnostic contribution that identifies and quantifies the fundamental scaling limitation of the bundling process. This is not a method or a positive result, but a characterization of when and why the approach fails—which is equally important for practitioners.

The core insight: when multiple SNNK layers are bundled, the random feature approximation error from each layer propagates forward and compounds. The paper formalizes this through a Lipschitz continuity argument (Appendix B): if the kernel function K is δ-Lipschitz in its input arguments (where δ(a) → 0 as a → 0), then the probability that the d-layer bundled network's output deviates from the exact network's output by more than ε + δ(ε) + δ(δ(ε)) + ... (a composition of d-1 δ-functions) is bounded by d times the single-layer concentration bound from Azuma's inequality (Equation 18). The dependence on the boundedness of the random features (guaranteed when A < 0 in the Λ_g construction, per Remark 3.1) is explicit in the exponential concentration term exp(-ε^2 / (8mc^2)).

Why this is a diagnostic concept rather than just an error bound: it tells the practitioner what controls the accuracy-efficiency tradeoff when bundling multiple layers. The number of random features m needed for a given accuracy grows with three factors: (1) the depth d of the bundled stack (linearly, through the union bound), (2) the Lipschitz constant L of the kernel function (through how quickly δ(a) decays—a larger L means more error amplification per layer), and (3) the bound c on the feature values (through the concentration exponent). Conversely, for a fixed m, there is a limit on how many layers can be bundled before the accumulated error becomes unacceptable.

This explains the empirical observation in the uptraining experiments (Appendix J.4, Figures 12–13) that performance degrades gracefully rather than catastrophically as more FFN blocks are replaced with bundled SNNKs. With 12 layers bundled, accuracy drops modestly; the error accumulates but does not explode. The Lipschitz analysis predicts this: if each layer's approximation error is ε and the kernel is sufficiently smooth (small δ), the total error grows roughly linearly with depth, not exponentially.

The diagnostic value extends to characterizing which activation functions are amenable to deep bundling. Activations with smoother Fourier transforms (smaller high-frequency components) produce kernel functions with smaller Lipschitz constants, allowing deeper bundling at fixed m. Conversely, activations with sharp features (e.g., ReLU's non-differentiability at zero) produce larger Lipschitz constants and limit bundling depth. This gives a principled criterion for choosing activations in SNNK-based architectures—one that the paper does not explicitly develop but that its analysis enables.

This innovation is an analytical contribution: it transforms the empirical question "how many layers can I bundle?" into a theoretically grounded tradeoff involving m, depth, and activation smoothness. For practitioners, this replaces trial-and-error with a diagnostic framework for anticipating and controlling approximation error in bundled architectures.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses a wide range of datasets spanning synthetic kernel estimation to real-world NLP and vision benchmarks. For pointwise kernel estimation (Section 4.1), synthetic data is generated with $d = 2000$, $l = 1$, and inputs/weights drawn from $\frac{1}{\sqrt{d}} \text{Unif}(0, 1)$. Toy experiments (Section 4.2) use a 2-body quantum physics simulation (QPNN), a natural image for SIREN-based fitting, and MNIST (LeCun & Cortes, 2010) for digit classification (60K training, 10K test across 10 classes). For NLP fine-tuning (Section 4.3), the paper uses the GLUE benchmark (Wang et al., 2018), consisting of 8 natural language understanding tasks: RTE, MRPC, QNLI, QQP, SST-2, MNLI, STSB, and CoLA, with dataset sizes ranging from 2.49K (RTE) to 393K (MNLI) training examples (detailed statistics in Table 2). Vision fine-tuning experiments use CIFAR-10 (50K training, 10K test, 10 classes), CIFAR-100 (50K training, 10K test, 100 classes) (Krizhevsky et al., 2009), and ImageNet-1k (Deng et al., 2009). For UCI experiments (Section 4.5), the paper uses three large-scale classification datasets: CoverType2 (~510K points, $d = 54$), HIGGS (~11M points, $d = 28$), and HEPMASS (~11M points, $d = 28$). For uptraining experiments (Appendix J.4), the paper evaluates on GLUE, CIFAR-10, and CIFAR-100.

• Base model(s). For NLP experiments, the paper uses BERT-base (Devlin et al., 2019) with 110M parameters and hidden dimension $d = 768$. For vision experiments, it uses ViT-base (Kolesnikov et al., 2021) pretrained on ImageNet-21k, specifically the checkpoint google/vit-base-patch16-224-in21k, with 86M parameters and $d = 768$. For toy experiments, the base models are: a SIREN network (Sitzmann et al., 2020) with 3 layers and hidden dimension $256 \times 256$, a QPNN network (Sehanobish et al., 2021) with hidden layer size $128 \times 128$, and a 3-layer MLP with hidden dimension $512 \times 512$ for MNIST. For UCI experiments, a 3-layer MLP with dropout and ReLU activations serves as baseline, with the first two layers being ReLU(Linear($d$, 512)) and ReLU(Linear(512, dparam)), where dparam is varied. The paper does not explicitly justify why BERT-base and ViT-base were chosen beyond their status as widely-used representative architectures for text and vision Transformers respectively.

• Metrics. For pointwise kernel estimation, the paper reports the mean relative error of the NN output (averaged over $s = 500$ instantiations of the RF mechanism) and the empirical standard deviation as a function of the number of random projections. For GLUE tasks, the paper follows standard practice: MCC (Matthews correlation coefficient) for CoLA, F1 score for MRPC and QQP, Spearman correlation for STSB, and accuracy for RTE, QNLI, SST-2, and MNLI. For image classification, accuracy is reported. For toy experiments, accuracy (MNIST) and visual quality of reconstruction/comparison to ground truth (SIREN image fitting, QPNN potential learning) are used. For UCI experiments, cross-entropy loss is reported as a function of training parameter count. For uptraining experiments, accuracy or task-specific metrics are plotted against model size (MB) and inference FLOPs to show the accuracy-efficiency tradeoff. All GLUE results are averaged over 5 random seeds (stated in Table 1 caption).

• Baselines. The paper compares against a substantial set of baselines across different experimental settings:

  • Pooler linearization (Section 4.3.1, Table 1 top half): The primary baseline is BERT-base with frozen Transformer layers and tuned pooler + classifier (Lee et al., 2019). This is a standard fine-tuning approach where only the final pooler layer (linear + tanh) and classification head are trained.
  • Adapter experiments on GLUE (Section 4.3.2, Table 1 bottom half): Three baselines are used: (1) Adapter-baseline (Moosavi et al., 2022) using the Houlsby configuration with adapter size 48; (2) Adaptable Adapter (AA) (Moosavi et al., 2022), which includes additional gating and dropping mechanisms; (3) For broader context, Table 8 compares against Full Finetune, Adapter (Houlsby) with size 48 (Houlsby et al., 2019), Adapter (Pfeiffer) (Pfeiffer et al., 2020), LoRA with rank decomposition (Hu et al., 2022), and BitFit (Zaken et al., 2022).
  • Adapter experiments on Vision (Table 7): Comparisons include Full Finetune, BitFit, Adapter (Houlsby), Adapter (Chen et al., 2022), AdapterDrop (Rücklé et al., 2021), LoRA, Transformer-probing, LoRA-Fix, LayerNorm Tuning, Attention Tuning, LePE Tuning, RPB Tuning, KAdaptation (He et al., 2022b), Visual Prompt Tuning (Jia et al., 2022), and AdaptFormer-64 (Chen et al., 2022).
  • Bundled network experiments (Appendix J.3, Table 5): Baselines are Pooler + Classifier (Lee et al., 2019) and Linear Probe (where only the classifier head is trained, pooler frozen).
  • UCI experiments (Section 4.5): The baseline is a 3-layer MLP with varying middle-layer output dimension dparam.
  • Uptraining experiments (Appendix J.4): The baseline is full fine-tuning of BERT-base and ViT-base (sourced from Pfeiffer et al., 2020 for BERT).
  • Toy experiments: The baseline neural networks are the original SIREN, QPNN, and 3-layer MLP architectures without SNNK replacements.

• Generation budget / compute accounting. The paper uses several distinct metrics of computational cost depending on context:

  • Number of random features ($m$): For SNNK layers, $m$ is the primary efficiency knob. The paper uses $m = 32$ for QPNN and MNIST, $m = 64$ for SIREN image fitting, $m = 16$ for all adapter experiments (32 for ImageNet), $m = 8$ for uptraining experiments, and sweeps $m$ across various values for ablation studies (Figures 5, 8, 9). For pooler linearization, the number of random features is also varied in ablations (Figure 8).
  • Number of trainable parameters: This is the primary compression metric. The paper reports parameter counts for adapter configurations: the ReLU-SNNK-Adapter uses only 0.3M trainable parameters vs. 0.9M for the Adapter-baseline (Table 8) vs. 110M for full fine-tuning. For vision adapters, the same 0.3M figure is reported (Table 7). For bundled pooler+classifier, ~2K parameters are trained vs. ~59K for the baseline (Table 5).
  • Model size and FLOPs (uptraining experiments): The paper reports model size in MB and inference FLOPs in billions for bundled Transformer variants (Table 6, Figures 12-13). For example, bundling 6 FFN blocks reduces BERT size from 440 MB to 226.71 MB and inference FLOPs from 716B to 484B.
  • Timing: For uptraining, the paper reports wall-clock inference time: SNNK-BERT takes 0.32 seconds per batch of 64 (sequence length 128) vs. 0.44 seconds for original BERT; SNNK-ViT takes 0.26 seconds per batch of 32 images vs. 0.33 seconds for ViT (Appendix J.4).
  • The paper does not account for the one-time cost of computing random features or the cost of the initial Gaussian projection matrix $G$ in parameter budgets—these are fixed and can be precomputed.

• Cross-validation / statistical protocol. For GLUE experiments, all reported numbers are averaged over 5 random seeds (stated in Table 1 caption). For MNIST and image classification experiments, a 25% stratified random sampling from the training set is used to create a validation set for model selection; the best model on this validation set is then evaluated on the holdout test set (Appendix F). For pointwise kernel estimation, results are averaged over $s = 500$ instantiations of the RF mechanism to quantify estimator variance. For UCI experiments, no explicit cross-validation protocol is stated—the paper reports cross-entropy loss curves comparing SNNK to MLP baselines of varying sizes (Figure 6). For uptraining experiments, the paper trains for 5-20 epochs depending on dataset and selects the best model based on validation loss (Appendix H.6). The paper does not report confidence intervals or statistical significance tests for any of its comparative results, which is a notable omission given the relatively small GLUE dev sets (e.g., RTE has only 277 evaluation examples).

Main Quantitative Results

Pointwise Kernel Estimation: URF Accuracy with Small $m \ll d$

The paper verifies that the URF mechanism provides accurate pointwise kernel estimation even with substantially fewer random projections than the input dimension. For a SIREN-FFL with $\sin$ activation and $d = 2000$, the mean relative error drops rapidly as the number of random projections $p$ increases (Figure 2g). The paper reports that "our SNNK provided an accurate approximation with a much smaller number of random projections than the dimensionality $d$ of the input vectors" (Section 4.1). The standard deviations are reported as extremely small—ranging from $2.5 \times 10^{-9}$ to $1 \times 10^{-11}$ (Figure 2 caption)—though the paper acknowledges that the maximum $p$ for the SIREN case is larger than for the arc-cosine case because SIREN "in theory produces larger variance per random projection." For the arc-cosine-FFL (Figure 2h), similar behavior is observed, with standard deviations in the range $1 \times 10^{-12}$ to $3 \times 10^{-8}$. These results validate the unbiasedness and low-variance properties of the URF mechanism in a controlled synthetic setting, but do not directly predict downstream network performance where approximation errors interact with learning dynamics.

Toy Experiments: SNNKs as Drop-in FFL Replacements Preserve Function While Reducing Parameters

The SIREN image fitting experiment (Figure 3, rightmost three columns) demonstrates that replacing the hidden layer of a 3-layer SIREN network ($256 \times 256$ hidden matrices) with a sine-variant SNNK using only 64 random features produces an "accurate approximation" of the original network's output, including both the image reconstruction and its derivatives. The visual comparison in Figure 3 shows the SNNK-augmented SIREN (bottom row) producing images and gradient visualizations that closely match the original SIREN (top row). The paper notes that cosine-variant SNNK "performs similarly in this task" (Appendix G.1.2).

The QPNN experiment (Figure 3, left column) shows that replacing the $128 \times 128$ hidden layer of QPNN with an SNNK layer using 32 random features "can learn the potential function perfectly even using less trainable parameters than the baseline QPNN." The three panels (top to bottom: ground truth potential, learned by QPNN, learned by QPNN-SNNK) show visually indistinguishable results. Both sine and cosine SNNK variants reportedly perform similarly (Appendix G.1.1).

On MNIST (Figure 9, left and middle), the SNNK-MLP with 32 random features is reported to "match or outperform" the baseline 3-layer MLP with $512 \times 512$ hidden layers. However, the paper does not report the specific accuracy numbers in the main text—only the trends across different random feature counts are shown in Figure 9 (left). Training curves (Figure 9, middle) show "lower training losses and stable training for SNNK-MLP across different number of random features." The comparison of trainable parameters between SNNK and baseline architectures for all toy experiments is summarized in Figure 7.

Pooler Layer Linearization: Competitive GLUE Performance with 1/1000 Storage Compression

Table 1 (top half) reports that the Cosine-SNNK-pooler model outperforms the BERT-baseline (Lee et al., 2019) on 5 of 8 GLUE tasks: RTE (61.36 ± 1.15 vs. 57.5, +3.86 points), MRPC (82.07 ± 1.07 vs. 81.5, +0.57), SST-2 (85.21 ± 0.34 vs. 84.9, +0.31), STSB (78.93 ± 0.37 vs. 78.1, +0.83), and CoLA (35.81 ± 0.96 vs. 29.4, +6.41). It underperforms on QNLI (73.5 vs. 74.5, -1.0), QQP (70.43 vs. 72.0, -1.57), and MNLI (52.69 vs. 56.4, -3.71). The paper emphasizes the storage savings: after bundling the linearized pooler with the classifier, the combined weight matrix of size (number of random features × number of classes) replaces the original pooler + classifier storing, reducing storage from 18.92 Megabit to 0.02 Megabit—a 1/1000 compression factor (Appendix E).

Ablation studies on the number of random features (Figure 8) show that performance is generally stable across a range of $m$ values for most tasks, though the paper does not report the exact sweep range. Notably, the paper finds that starting with pretrained pooler weights (Figure 9, right) "produces better results on a wide range of text and image datasets" compared to randomly initialized linearized layers, confirming that the initial feature quality from pretrained weights matters.

The paper also reports a negative result: attempting to linearize the tanh pooler activation using brute-force polynomial kernel methods (Appendix L.1, Table 3) produces "poor performance"—RTE drops to 52.08, MRPC to 68.71, STSB to 63.49, and CoLA to 20.72. The paper attributes this to the "sparse" features produced by this method distorting the pretrained representations, motivating the need for the URF approach.

SNNK-Inspired Adapter Layers: ~5× Parameter Reduction with Competitive or Superior Accuracy

GLUE results (Table 1, bottom half): The ReLU-SNNK-Adapter, using only 16 random features and 0.3M trainable parameters (vs. 0.9M for the Adapter-baseline), outperforms the Adapter-baseline (Moosavi et al., 2022) on 5 of 8 tasks: RTE (69.68 ± 1.24 vs. 63.83, +5.85), MRPC (91.26 ± 1.39 vs. 84.8, +6.46), QNLI (90.44 ± 0.16 vs. 90.63, comparable), SST-2 (92.31 ± 0.27 vs. 91.74, +0.57), and CoLA (58.21 ± 0.63 vs. 56.51, +1.70). It underperforms on QQP (85.82 vs. 88.12, -2.30), MNLI (82.06 vs. 83.53, -1.47), and STSB (88.81 vs. 88.48, comparable). Compared to the Adaptable Adapter (AA), the ReLU-SNNK-Adapter outperforms on 7 of 8 tasks, with the exception being QQP. The Cosine-SNNK-Adapter variant (Table 4) performs moderately worse overall—beating the baseline on 4 of 8 tasks (RTE, MRPC, SST-2, STSB) while using the same parameter budget.

Ablation on random features (Figure 5): Increasing the number of random features for the ReLU-SNNK-Adapter generally improves performance on larger datasets. The paper notes: "we can improve performance and match the baselines on large datasets (ex. MNLI, QNLI) as we increase the number of random features." This is consistent with the observation in Houlsby et al. (2019) that larger adapter sizes help on MNLI. The paper reports that using 16 random features already achieves strong results, with further improvements from increasing $m$.

Vision adapter results (Figure 4, right 3 panels; Table 7): The ReLU-SNNK-Adapter achieves 98.87% on CIFAR-10 (vs. 99.1% for Houlsby adapter with 1.8M parameters, vs. 98.95% for full fine-tuning with 85M parameters) and 92.11% on CIFAR-100 (vs. 91.2% for Houlsby adapter, vs. 91.67% for full fine-tuning), while using only 0.3M trainable parameters. Compared to the full sweep of parameter-efficient methods in Table 7, SNNK-Adapter is competitive with or outperforms methods using comparable or larger parameter budgets (e.g., LoRA at 0.22M: 98.7 on CIFAR-10, 90.6 on CIFAR-100; KAdaptation at 0.11M: 97.9 on CIFAR-10, 91.2 on CIFAR-100). A variant with a frozen pooler layer (ReLU-SNNK-Adapter*) achieves 98.87 and 92.11 respectively.

Table 8 comparison with broader PEFT methods: Against LoRA (0.3M parameters), the ReLU-SNNK-Adapter is competitive but generally slightly behind—LoRA achieves higher scores on MRPC (90.1 vs. 91.26 actually favors SNNK), QQP (88.5 vs. 85.82), MNLI (84.1 vs. 82.06), and STSB (89.1 vs. 88.81). Against BitFit (0.1M parameters), SNNK-Adapter outperforms on RTE (though BitFit achieves 72.3 vs. 69.68, SNNK wins on MRPC), QNLI (90.44 vs. 90.2), QQP (85.82 vs. 84.0), SST-2 (92.31 vs. 92.1), and CoLA (58.21 vs. 58.8), while trailing on MNLI and STSB.

Bundled Network Training: Proof-of-Concept for Closed-Form Learning with Minimal Parameters

Table 5 (Appendix J.3) reports results for training a fully bundled pooler + classifier layer with 1024 random features, yielding only ~2K trainable parameters. The Bundled-final-layers model achieves: RTE 60.01 (vs. 57.5 for Pooler+Classifier baseline and 57.76 for Linear Probe), MRPC 81.48 (vs. 81.5 and 81.23), QNLI 72.21 (vs. 74.5 and 69.18), QQP 70.19 (vs. 72.0 and 62.22), SST-2 82.97 (vs. 84.9 and 83.02), MNLI 53.69 (vs. 56.4 and 43.4), STSB 71.84 (vs. 78.1 and 69.87), CoLA 32.79 (vs. 29.4 and 35.41). The bundled model substantially outperforms the Linear Probe on QNLI, QQP, and MNLI (by 3.03, 7.97, and 10.29 points respectively), demonstrating that the nonlinear feature preprocessing adds meaningful capacity beyond linear classification. On the STSB regression task, the paper reports achieving a Spearman correlation of 67 using the closed-form least-squares solution "in barely a few seconds on a CPU" (Appendix J.3).

For vision, bundled final layers achieve 95.8 on CIFAR-10 and 82.73 on CIFAR-100, "closely matching the baseline results" (Appendix J.3, no baseline numbers provided for direct comparison in the text).

Uptraining Transformers: 40-50% Model Size Reduction with Minimal Accuracy Impact

Table 6 quantifies the compression achieved by replacing top-$k$ FFN blocks of BERT and ViT with SNNK layers (8 random features) followed by bundling. For BERT: replacing 6 layers reduces training parameters from 110M to 72M, inference parameters from 110M to 69M (37% reduction), training FLOPs from 716B to 601B, and inference FLOPs from 716B to 484B (32% reduction). Model size drops from 440 MB to 226.71 MB (48% reduction). For ViT: with 6 layers bundled, training parameters reduce from 86M to 48M, inference parameters from 86M to 46M (47% reduction), training FLOPs from 563B to 475B, and inference FLOPs from 563B to 384B (32% reduction). Model size drops from 346 MB to 176.42 MB (49% reduction).

Figures 12 and 13 show the accuracy-model size and accuracy-FLOPs tradeoff curves for BERT on GLUE tasks and ViT on CIFAR-10/100. The paper states that the reduction in model size and FLOPs "comes with only a slight decrease in accuracy" (Appendix J.4). However, the paper does not report specific accuracy numbers in either the main text or Appendix J.4 for these uptraining experiments—only the curves in Figures 12 and 13 are provided, making it impossible to quote exact accuracy degradation from the text. The paper notes that inference wall-clock time drops from 0.44 seconds to 0.32 seconds per batch for BERT and from 0.33 to 0.26 seconds for ViT on a free Google Colab GPU (Appendix J.4).

UCI Experiments: Matching or Outperforming MLPs at a Fraction of Parameters

Figure 6 shows cross-entropy loss curves on three UCI datasets (CoverType2, HIGGS, HEPMASS) for SNNK-replaced middle layers vs. MLP baselines of varying middle-layer sizes. The paper reports: "SNNK matches or outperforms the baseline while using only a fraction of the training parameters." The x-axis of Figure 6 is training parameter count, making this a direct efficiency comparison. However, specific loss values are not quoted in the text—the claim is based on visual inspection of the curves. The paper does not specify the exact parameter counts for SNNK vs. each MLP size.

Ablation Studies and Robustness Checks

Number of random features for SNNK-pooler (Appendix I, Figure 8): The Cosine-SNNK-pooler performance is evaluated across varying numbers of random features on all GLUE tasks. The paper does not report the specific range swept, but the curves in Figure 8 show that performance is relatively stable across $m$ values for most tasks, with the largest datasets (MNLI, QNLI, QQP) showing some sensitivity. The BERT-Pooler baseline (from Lee et al., 2019) is plotted as a horizontal reference line.

Number of random features for SNNK-Adapter (Figure 5): Performance generally improves with increasing $m$ on larger datasets (MNLI, QNLI, QQP), while smaller datasets (RTE, MRPC, CoLA, STSB, SST-2) are less sensitive. The paper explicitly connects this to the Houlsby et al. (2019) observation that larger adapter sizes help on MNLI.

Number of random features for MNIST (Figure 9, left and middle): The SNNK-MLP is trained with varying $m$ for 25 epochs. The paper states that SNNK-MLP achieves competitive or better accuracy than the baseline MLP across different random feature counts, with "lower training losses and stable training" (Figure 9, middle). However, specific accuracy numbers are not reported in the text for these runs.

Starting with pretrained vs. random pooler weights (Figure 9, right): The paper compares initializing the SNNK-pooler from pretrained BERT pooler weights vs. random initialization, finding that "the initial features coming from the pretrained pooler weights produce better results on a wide range of text and image datasets." This confirms that while the SNNK formulation can theoretically be trained from scratch, leveraging pretrained features provides a meaningful warm start.

Sine vs. cosine SNNK variants for toy experiments (Appendix G.1): For QPNN, both sine and cosine SNNK variants produce similar performance. For SIREN image fitting, both variants accurately model the signal and its derivatives, which the paper notes is unsurprising since "cosine function is a shifted sine function and so they behave similarly." This suggests some robustness to the choice of periodic activation for these physics-inspired applications.

Cosine-SNNK-Adapter vs. ReLU-SNNK-Adapter (Table 4 vs. Table 1 bottom half): The Cosine-SNNK-Adapter performs worse than the ReLU-SNNK-Adapter on most GLUE tasks (beats baseline on 4 of 8 vs. 5 of 8), though both use the same 0.3M parameter budget. Specific comparisons: CoLA drops from 58.21 (ReLU) to 52.12 (Cosine), MNLI from 82.06 to 79.28, QNLI from 90.44 to 88.67. The paper does not discuss why ReLU outperforms Cosine in this setting, though the ReLU-SNNK's connection to arc-cosine kernels and its proven greater expressiveness than standard FFLs (Theorem 3.2, Remark 3.3) may be a factor.

Tanh-SNNK via brute-force polynomial linearization (Appendix L.1, Table 3): This is a clear negative result. The Tanh-SNNK-pooler achieves substantially worse performance than the BERT-baseline on all tested tasks: RTE 52.08 vs. 57.5, MRPC 68.71 vs. 81.5, STSB 63.49 vs. 78.1, CoLA 20.72 vs. 29.4. The paper attributes this to the "extremely sparse kernels" produced by the polynomial method, which "distort" the pretrained features. This negative result justifies the development of the URF mechanism as a superior alternative for non-positive, non-symmetric activation functions.

Bundling depth (number of FFN layers replaced) in uptraining (Figures 12-13): While not presented as a formal ablation study, the uptraining experiments show the effect of progressively bundling more Transformer layers (from 1 to 12). The curves in Figures 12 and 13 demonstrate that accuracy degrades gradually rather than catastrophically as more layers are bundled, consistent with the error propagation analysis in Appendix B which predicts roughly linear error accumulation with depth.

Gating vector initialization (Appendix D): The paper notes that initializing the gating vector $v$ to zero is critical for stable adapter training—it ensures the SNNK-adapter block behaves as the identity at initialization. Alternative initialization schemes (Gaussian with small variance, or setting the weight matrix $A$ to zero) are mentioned as having been tried but found to cause optimization difficulties. However, this is presented as a design choice justification rather than a formal ablation with comparative results.

Choice of adapter configuration (Appendix G.3): The paper uses the Houlsby configuration (adapters added after attention and after FFN) for all adapter experiments, noting that for simplicity they "choose to work with the original Houlsby configuration in this work" rather than exploring Pfeiffer or other variants. Adapter size 48 is used for all baseline experiments.

Critical Assessment

This section evaluates whether the experimental evidence genuinely supports the paper's central claims, identifies specific weaknesses and gaps, and surfaces experiments that would have strengthened the paper but were not run.

Claim: SNNKs can replace regular FFLs with favorable computational properties (reduced parameters and inference cost)

What the experiments demonstrate: This claim is supported across a range of scales, but with important nuances. The toy experiments (SIREN, QPNN, MNIST) show that SNNKs can replace hidden layers in small networks with 32-64 random features and match performance, while Figure 7 confirms parameter reduction. The pooler experiments show that the pooler layer specifically can be linearized with competitive GLUE results. The adapter experiments provide the strongest evidence: ReLU-SNNK-Adapter with 0.3M parameters matches or exceeds 0.9M-parameter baselines on most GLUE and vision tasks. The uptraining experiments show 40-50% model size reduction with minimal accuracy impact.

What is not demonstrated: The paper never replaces an entire feedforward network from scratch with SNNKs and trains it to match the original—the SIREN and QPNN experiments replace only one hidden layer, and the adapter experiments add SNNKs as auxiliary modules to frozen pretrained networks rather than replacing the original FFN blocks. The uptraining experiments come closest by replacing FFN blocks, but these are fine-tuning experiments starting from pretrained weights, not training from scratch. Whether SNNKs can serve as drop-in replacements for FFLs in randomly initialized deep networks trained end-to-end remains untested.

The pooler experiments have an important confound: The Cosine-SNNK-pooler uses a cosine activation while the original pooler uses tanh. The paper acknowledges this (Appendix G.2): "Both the cosine and the tanh functions have the same range of values so we find no instabilities or difficulties in training." However, this means the SNNK-pooler is not an approximation of the tanh pooler—it is a different function that happens to work well. The Tanh-SNNK attempt (Table 3) actually failed. So the pooler experiments demonstrate that a cosine-based SNNK can serve as a pooler, not that SNNKs successfully approximate arbitrary FFLs. This is a weaker claim.

Claim: The URF mechanism provides unbiased estimation of f(w^T x + b) for any activation with a well-defined Fourier Transform

What the experiments demonstrate: The pointwise kernel estimation experiments (Figure 2g,h) show that for sine and arc-cosine kernels, the RF-based estimator converges to the true value with very low variance as $m$ increases. The mean relative error decreases and standard deviations are tiny ($10^{-8}$ to $10^{-12}$).

What is not demonstrated: The paper claims universality—that URFs work for "any $f$ with a well-defined Fourier Transform"—but only tests sine and cosine (which have trivial distributional Fourier transforms with two Dirac atoms) and ReLU (indirectly, via the arc-cosine kernel rather than the Fourier-based URF). The ReLU case is particularly important because ReLU's Fourier transform is "not well-behaved" (Appendix N), requiring smoothing/truncation that "incurs an error." The paper does not empirically evaluate the URF mechanism for ReLU, GELU, tanh, sigmoid, or any other common activation besides sine/cosine. The tanh linearization attempt using the alternative polynomial method failed (Table 3), but the paper never reports results for a Fourier-based URF tanh-SNNK. This is a significant gap between the theoretical claim of universality and the empirical coverage.

The ReLU-SNNK layer does not use URFs: The ReLU-SNNK (Section 3.2) is constructed via the arc-cosine kernel connection (Theorem 3.2), not via the Fourier/URF mechanism. So the adapter experiments—the paper's main practical results—use a construction that is mathematically distinct from the URF derivation that occupies much of Section 3.1. The paper's two main technical contributions (URFs and ReLU-SNNK) are independent; the empirical success of one does not validate the other.

Claim: SNNKs are strictly more expressive than regular FFLs

What the experiments demonstrate: The theoretical claim is solid: Theorem 3.2 and Remark 3.3 prove that the ReLU-SNNK computes a function that cannot be expressed as $f(w^T x + b)$. The experimental evidence is indirect: ReLU-SNNK-Adapter outperforms standard adapters (which are bottleneck MLPs structurally similar to small FFLs) on 5 of 8 GLUE tasks. This is consistent with greater expressiveness, but could also be explained by other factors (the specific random projection pattern, the gating mechanism, favorable optimization dynamics).

What is not demonstrated: The paper never runs an experiment that isolates expressiveness as the causal factor. A direct test would be: compare a ReLU-SNNK layer against the best possible FFL of the same parameter count on a task designed to require angle/norm-sensitive computation that arc-cosine kernels capture but dot-product-based neurons cannot. Without such a controlled experiment, the claim of "strictly more expressive" remains a theoretical property of the architecture whose practical benefits are plausible but not directly verified. The adapter results are suggestive but confounded by the many other differences between SNNK adapters and standard adapters (random projection structure, gating vector, different optimization landscape).

Claim: The bundling process leads to additional compression gains, and in the extreme case, optimal parameters can be expressed via explicit formulae

What the experiments demonstrate: The bundled pooler+classifier experiments (Table 5) show that a ~2K-parameter model can approach the performance of a ~59K-parameter model on some GLUE tasks (RTE: 60.01 vs. 57.5, MRPC: 81.48 vs. 81.5). The closed-form solution on STSB (Spearman 67) demonstrates the explicit formula claim. The uptraining experiments show that bundling multiple FFN blocks achieves substantial size reduction with modest accuracy loss.

What is not demonstrated: The bundling experiments are limited in scope:

• The bundled network (Table 5) uses 1024 random features—far more than the 16-64 used in other experiments—and still underperforms the full pooler+classifier baseline on 5 of 8 tasks. The paper characterizes this as "competitive," but the gaps are non-trivial on QNLI (72.21 vs. 74.5), QQP (70.19 vs. 72.0), SST-2 (82.97 vs. 84.9), MNLI (53.69 vs. 56.4), and especially STSB (71.84 vs. 78.1)—a 6.26 point regression task drop. The 1024 random features partially defeat the purpose of compression.
• The fully bundled network from Equation 14—collapsing an entire deep network to $y = Wx$—is never empirically tested on a deep architecture. The pooler+classifier bundling collapses only the final two layers. The uptraining experiments bundle the FFN blocks but keep the attention layers intact. The extreme case that "opens a possibility to bypass backpropagation" remains a theoretical possibility without empirical demonstration at scale.

Claim: SNNKs provide up to 5× reduction in the number of trainable parameters while maintaining competitive accuracy

What the experiments demonstrate: The number "5×" appears to come from the adapter experiments: 0.3M SNNK-Adapter parameters vs. ~1.5-1.8M for standard adapters (Houlsby), and vs. 110M for full fine-tuning on BERT. The specific compression ratios vary:

• vs. Adapter-baseline (0.9M): ~3× reduction (Table 8)
• vs. Adapter (Houlsby, 1.8M): ~6× reduction
• vs. Full fine-tuning (110M): ~366× reduction
• Pooler storage: ~1000× reduction (18.92 Megabit to 0.02 Megabit)
• Uptraining: 40-50% model size reduction

The "5×" figure is a reasonable summary, though it undersells some comparisons and oversells others. The paper consistently achieves substantial parameter reduction, and on many (not all) tasks, accuracy is competitive or better.

A critical caveat on parameter counting: The paper counts only trainable parameters. The fixed random projection matrices (e.g., the Gaussian matrix $G$ in ReLU-SNNK) are not counted because they are not trained. However, they still need to be stored and applied. For the ReLU-SNNK-Adapter with $m = 16$ and $d = 768$, the Gaussian matrix $G \in R^{16 \times 768}$ contains 12,288 entries that must be stored and multiplied. These are not trained, but they contribute to memory and compute. The paper does not add these to the parameter budget, which is defensible for inference (they can be generated on-the-fly from a seed) but misleading for a strict memory comparison. In practice, one would either store $G$ (adding 12K parameters) or regenerate it from a fixed random seed (adding latency). Neither cost is accounted for.

Critical omissions and missing experiments

No statistical significance testing: None of the comparative results in Tables 1, 4, 5, 7, or 8 are accompanied by confidence intervals, hypothesis tests, or measures of statistical significance. The standard deviations reported in Table 1 are helpful, but the paper never states whether the differences between SNNK and baselines are statistically significant given the test set sizes (e.g., RTE has only 277 dev examples). For the headline claim that SNNK-Adapter "outperforms the baseline on 5 out of 8 datasets," it is unclear whether the 4-6 point improvements on RTE and MRPC are reliably above noise given the small dev sets.

No comparison against adapters with matched parameter counts: The ReLU-SNNK-Adapter uses 0.3M parameters. The baselines use 0.9M (Adapter-baseline), 1.8M (Houlsby), or 0.3M (LoRA). A fairer comparison would configure the standard adapters to use exactly 0.3M parameters (by reducing the bottleneck dimension) and compare at equal budget. The paper instead compares against adapters with larger parameter budgets, which makes the SNNK efficiency gains look more dramatic than an iso-parameter comparison would show. The LoRA comparison at 0.3M (Table 8) partially addresses this, and LoRA outperforms SNNK-Adapter on several tasks—suggesting that at matched parameter counts, the advantage is not clear-cut.

No results for Fourier-based URF SNNKs on real tasks: The two main practical successes—pooler linearization and adapter layers—use Cosine-SNNK and ReLU-SNNK respectively. Neither uses the full Fourier/URF mechanism from Section 3.1. The Cosine-SNNK uses the simplified two-atom construction for trigonometric functions; the ReLU-SNNK uses the arc-cosine kernel construction. The paper's most theoretically elaborate contribution (URFs with four-component Fourier decomposition, importance sampling, softmax kernel linearization) is empirically validated only on the pointwise kernel estimation toy experiment (Figure 2g). Whether this mechanism works for non-trigonometric activations (tanh, sigmoid, GELU, Swish) in the context of training real networks is entirely untested.

The uptraining experiments lack specific accuracy numbers: Figures 12 and 13 show curves, but the paper does not report the actual accuracy values before and after bundling. Statements like "minimal impact on accuracy" are qualitative and cannot be verified from the text alone. The reader must estimate accuracy degradation by visually reading off the plots.

No comparator against other SNNK variants for the same task: The adapter experiments use ReLU-SNNK; the pooler experiments use Cosine-SNNK. The paper does not systematically compare ReLU vs. Cosine vs. Fourier-based SNNKs on the same task to determine which construction works best in which setting. The one exception is Table 4 vs. Table 1 bottom half (Cosine-SNNK-Adapter vs. ReLU-SNNK-Adapter), where ReLU is clearly superior—but this comparison is incidental, not systematic.

Single model families: All GLUE experiments use BERT-base-uncased; all vision experiments use ViT-base. There are no experiments with BERT-large, RoBERTa, DeiT, Swin, or any other architecture. The generalizability of SNNK compression across model scales and families is assumed but untested.

No latency benchmarking in a deployment setting: The uptraining experiments report wall-clock time on a "free Google Colab" GPU (Appendix J.4). This is not a controlled latency measurement. The paper does not report throughput, latency percentiles, or memory bandwidth utilization in a production inference setting. The theoretical FLOP reductions (Table 6) are useful, but real-world speedups depend on hardware utilization, memory access patterns, and whether the operations map efficiently to GPU kernels.

Limited bundling depth ablation: The uptraining experiments bundle 1-12 layers, but the paper does not ablate the number of random features needed per layer as a function of bundling depth. The error propagation analysis (Appendix B) predicts that deeper bundling requires larger $m$ to maintain accuracy—this is not empirically verified. All uptraining uses $m = 8$, and the graceful degradation with depth (Figures 12-13) is observed but not systematically characterized.

Summary of conditional validity

• The claim that SNNKs provide parameter reduction with competitive accuracy holds most strongly for the adapter-based fine-tuning of frozen pretrained Transformers using ReLU-SNNK with $m = 16$ on GLUE and vision tasks. This is the paper's most robust empirical result.
• The claim that URF provides accurate kernel estimation holds for trigonometric activations on synthetic pointwise evaluation. Extension to other activations is theoretically supported but empirically unverified.
• The claim that bundling enables closed-form solutions holds in the limited case of bundling the final pooler and classifier layers of BERT. Full-network bundling remains untested.
• The claim of "strictly more expressive" is a theoretical property of the ReLU-SNNK architecture whose practical benefits are consistent with but not directly demonstrated by the adapter results.
• The uptraining results demonstrate that partial bundling of FFN blocks is viable, but specific accuracy numbers are not provided, making precise assessment impossible from the text alone.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

The assumption or constraint. The compute-optimal framework in the example paper requires estimating each prompt's difficulty before allocating the test-time budget. The method for doing so—generating 2048 samples for a prompt and averaging the verifier's correctness predictions—is extraordinarily expensive. The authors acknowledge they "did not experiment with PRM tree-search techniques in combination with revisions" (Section 8), but the more fundamental issue is flagged explicitly in the paper's own discussion of limitations: the difficulty estimation cost is not included in any budget calculation, making the reported 4× efficiency gains an upper bound that assumes free difficulty estimation.

The consequence. In a realistic deployment, the total cost equals difficulty estimation plus strategy execution. The estimation step (2048 samples) alone exceeds the largest test-time budgets studied (256–512 generations). For many practical settings, the estimation cost would dominate the total compute, potentially erasing the reported gains entirely or even making the approach more expensive than simply running best-of-N with the full budget. The 4× figure should therefore be understood as conditional on having a cheap difficulty oracle—which does not exist in the paper.

What evidence exists in the paper. The paper explicitly notes this gap: "our experiments do not account for this cost largely for simplicity" (Section 3.2). The pointwise kernel estimation experiments (Figure 2g,h) use m random features well below the input dimension d = 2000, but these are synthetic and do not involve the difficulty estimation pipeline. No experiment measures the amortized cost of difficulty estimation in a full deployment loop.

Mitigation status. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) and mentions adaptive difficulty estimation as a direction. No such model is developed or evaluated. Until this gap is closed, the practical efficiency gains remain hypothetical.

---

The URF Mechanism Is Empirically Validated Only for Trigonometric Activations—Not for the General Case It Claims

The assumption or constraint. The paper's most theoretically elaborate contribution—Universal Random Features (URFs) with four-component Fourier decomposition, importance sampling, and softmax kernel linearization (Section 3.1)—is designed for any activation function f with a well-defined Fourier Transform (classical or distributional). The paper states that "function K_f does not need to be positive semi-definite. This is critical for applications in neural networks, where the activation function f usually does not correspond to a positive semi-definite kernel" (Section 1). The claim is one of universality: the URF template should work for arbitrary activations.

The consequence. In practice, the paper's two main successes use constructions that bypass the full URF mechanism:

• The Cosine-SNNK-pooler (Section 4.3.1) exploits the trivial distributional Fourier transform of cosine (two Dirac delta atoms), which collapses the importance sampling to deterministic two-frequency evaluation.
• The ReLU-SNNK-Adapter (Section 4.3.2) uses the arc-cosine kernel construction from Theorem 3.2 (Cho & Saul, 2011), not Fourier-based URFs at all.

The paper never empirically evaluates a Fourier-based URF SNNK for ReLU, GELU, tanh, sigmoid, or any non-trigonometric activation in a real training setting. The tanh linearization attempt using polynomial kernel methods (not URFs) failed: "applying a sparse kernel distorts [the pretrained features]" (Appendix L.1, Table 3, with RTE dropping from 57.5 to 52.08). Whether the Fourier-based URF would succeed for tanh where the polynomial method failed is unknown. The theoretical universality of URFs is not in question, but their practical effectiveness for common activations beyond sine/cosine is entirely unverified.

What evidence exists in the paper. The pointwise kernel estimation experiment (Figure 2g) validates the URF mechanism for a SIREN-FFL with sine activation on synthetic data (d = 2000, l = 1). The paper notes that for ReLU, the Fourier transform is "not well-behaved" (Appendix N) and requires smoothing/truncation that "incurs an error." No experiment measures this error in a training context.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation or suggest empirical validation of URFs for non-trigonometric activations as future work. The gap between the theoretical claim (universality) and empirical coverage (sine/cosine only) is substantial and unremarked upon by the authors.

---

No Demonstration That SNNKs Can Replace FFLs When Training Deep Networks from Scratch

The assumption or constraint. The paper's central claim is that SNNKs can serve as "replacements of regular feedforward layers" (Section 1) with "favorable computational properties." This implies SNNKs should be viable not just as auxiliary modules or fine-tuning adapters, but as genuine architectural substitutes for FFLs in networks trained end-to-end from random initialization.

The consequence. All experiments where SNNKs perform well modify pretrained networks:

• The pooler experiments (Section 4.3.1) replace only the final pooler layer of a frozen BERT model, and train only the linearized pooler + classifier (roughly 59K parameters out of 110M).
• The adapter experiments (Section 4.3.2) inject SNNKs as small auxiliary modules (~0.3M parameters) into frozen pretrained Transformers (110M or 86M parameters frozen).
• The uptraining experiments (Appendix J.4) replace FFN blocks in pretrained BERT/ViT and fine-tune—they do not train from scratch.
• The toy experiments (SIREN, QPNN, MNIST) replace only one hidden layer, not the full stack.

No experiment trains a deep SNNK-based network from scratch (no pretrained weights) and compares it to an equivalent FFL-based network. The pretrained backbone provides the vast majority of the model's representational capacity; the SNNK modules contribute a small delta. Whether SNNKs can learn hierarchical representations from scratch—where every layer must be trained simultaneously with no pretrained initialization—is an open question. The error propagation analysis (Appendix B) suggests that when multiple SNNK layers are stacked, variance accumulates, and the number of random features needed grows with depth—but this is never tested in a from-scratch training regime.

What evidence exists in the paper. The MNIST experiment (Figure 9, left) trains a 3-layer SNNK-MLP from scratch, but only the middle layer is replaced with SNNK—the input and output layers remain standard FFLs. The paper reports that SNNK-MLP "match or outperform" the baseline, but the experiment replaces one layer, not the full network. The bundled pooler+classifier experiment (Table 5) removes the non-linear pooler entirely and replaces it with a linear layer acting on fixed random features—this is a from-scratch setting for the classification head, but the pretrained BERT features remain frozen.

Mitigation status. Not addressed. The paper does not discuss from-scratch training of deep SNNK architectures as a limitation or future work. The claim of "replacements of regular feedforward layers" is therefore supported only in the context of pretrained models or single-layer replacement, not for general deep network training.

---

Parameter Counting Omits the Cost of Fixed Random Projection Matrices

The assumption or constraint. Throughout the paper, the reported parameter counts for SNNK layers include only trainable parameters. The fixed random projection matrices (e.g., the Gaussian matrix G ∈ R^{l×d} in ReLU-SNNK, the Fourier frequency samples and Gaussian vectors in URF-based SNNKs) are excluded from parameter budgets because they are not learned—they are fixed after random initialization and shared across all inputs.

The consequence. These matrices must still be stored (occupying memory) and applied (consuming compute). For the ReLU-SNNK-Adapter with m = 16 random features and hidden dimension d = 768, the Gaussian projection matrix G contains 16 × 768 = 12,288 entries. The paper reports 0.3M trainable parameters for the SNNK-Adapter (Table 8). Adding G would increase this by about 4%—not enormous, but the gap widens for configurations with larger m or where multiple projection matrices are needed. The URF-based SNNKs require storing (or regenerating from seed) the Fourier frequency samples ξ_k, the Gaussian vectors g_k, and potentially the importance sampling parameters. The paper's headline compression numbers—particularly the pooler storage reduction from 18.92 Megabit to 0.02 Megabit (~1/1000)—exclude the random projection state entirely. While the random matrices can in principle be regenerated on-the-fly from a fixed seed rather than stored, this adds latency (regenerating random numbers) that substitutes for memory cost and is not included in any timing measurement.

What evidence exists in the paper. The paper never discusses the storage or compute cost of the fixed random projections. The uptraining experiments (Table 6) report parameter counts and FLOPs for bundled Transformers, but it is unclear whether the G matrices and URF state are included in these totals. The timing measurements (Appendix J.4: 0.32s vs. 0.44s for BERT) are end-to-end and would include any random regeneration cost—but this conflates compression savings with overhead in a way that makes it impossible to isolate the true benefit.

Mitigation status. The paper does not acknowledge this as a limitation. A more thorough accounting would report (1) trainable parameters, (2) total stored parameters (including fixed projections), and (3) a note on whether fixed projections are stored or regenerated in the timing experiments. This is a standard practice in efficient ML that the paper does not follow.

---

The Approach Is Not Demonstrated on Any Architecture Family Beyond BERT and ViT, Nor on Any Task Type Beyond Classification and Regression

The assumption or constraint. All SNNK experiments use either BERT-base (text) or ViT-base (vision) as the backbone architecture, and all tasks are either classification (GLUE, CIFAR, ImageNet, MNIST, UCI) or regression (STSB). The paper claims that SNNKs are a general-purpose "replacement of regular feedforward layers" (Section 1) that can "be used as a drop-in replacement for FFLs in a wide variety of NNs" (Section 4.2).

The consequence. Several important deployment scenarios are untested:

• Autoregressive generation (GPT-style decoder models, machine translation, summarization): In these architectures, the FFN blocks process variable-length sequences where each token's representation depends on all previous tokens. Whether SNNK compression degrades generation quality, causes error accumulation across decoding steps, or interacts badly with attention mechanisms in decoder-only architectures is unknown.
• Larger model scales (BERT-large, RoBERTa, DeiT, Swin, models with >300M parameters): The paper's experiments top out at 110M parameters (BERT-base). Whether the SNNK efficiency-accuracy tradeoff holds at larger scales—where FFN blocks constitute an even larger fraction of total parameters—is assumed but not shown. Scaling behavior of the random feature variance (which depends on d and m) could change in larger architectures.
• Multimodal models, speech, or reinforcement learning: The paper's tasks are all static supervised learning problems. In RL or sequential decision-making, the compounding effect of SNNK approximation variance over multiple timesteps could be substantial. The error propagation analysis (Appendix B) addresses depth but not temporal horizon.

The generalizability concern is heightened by the fact that the adapter experiments—the paper's strongest results—rely on a specific property of the fine-tuning setup: a large frozen pretrained backbone that already captures most of the task-relevant representation, with SNNKs contributing a small residual adaptation. This setup is particularly forgiving of approximation error in the SNNK modules. In a setting where the SNNK-replaced layers must carry more representational responsibility (e.g., a smaller model, or layers closer to the input), the accuracy impact could be larger.

What evidence exists in the paper. The paper reports results on 8 GLUE classification/regression tasks, 3 image classification benchmarks, and 3 UCI classification datasets. The toy experiments cover image reconstruction and physics simulation but are small-scale. No generation, sequential prediction, or RL task is evaluated. The paper does not discuss this scope limitation.

Mitigation status. Not addressed. The broad claims of generality are not accompanied by caveats about the tested task and architecture scope. Future work could extend evaluation to decoder-only architectures, larger models, and non-classification tasks—but the paper does not identify this as a priority.

---

The ReLU-SNNK Layer's Superiority Over Standard Adapters Is Not Isolated from Architectural Confounds

The assumption or constraint. The paper's headline adapter result (Table 1, bottom half) shows ReLU-SNNK-Adapter outperforming the Adapter-baseline on 5 of 8 GLUE tasks while using ~3× fewer trainable parameters (0.3M vs. 0.9M). The paper attributes this to SNNK's efficiency and, implicitly, the greater expressiveness of arc-cosine kernels. However, the SNNK-Adapter architecture differs from standard adapters in multiple ways beyond the core SNNK formulation:

• It includes a learned gating vector v (Appendix D) initialized to zero, which modulates the SNNK output element-wise and ensures identity-mapping behavior at initialization.
• The random projection pattern (Gaussian matrix applied to input and weights separately, followed by ReLU and dot-product) is structurally different from the low-rank bottleneck W_down → activation → W_up used in standard adapters.
• The training dynamics (optimization landscape, gradient flow) are different due to the random projection structure.

These confounds mean that the performance difference could be driven by the gating mechanism, the specific initialization scheme, or favorable optimization properties of the random projection structure, rather than by the arc-cosine kernel expressiveness or the two-tower disentanglement that the paper centers its narrative around. The paper does not ablate any of these architectural choices to isolate the contribution of the SNNK mechanism itself.

What evidence exists in the paper. The gating vector is described as necessary for stable training: "Other detailed initialization schemes are not studied and are beyond the scope of this work" (Appendix D). The paper reports a brief negative result: initializing the weight matrix A as zero or from a small-variance Gaussian "does not lead to good performance." However, this does not disentangle the gating vector's contribution from the SNNK formulation's contribution. The comparison between ReLU-SNNK-Adapter and Cosine-SNNK-Adapter (Table 4 vs. Table 1 bottom half) varies only the activation, but the cosine variant underperforms—suggesting the specific choice of ReLU + arc-cosine structure matters, but not isolating why (the activation nonlinearity, the kernel properties, or optimization behavior).

Mitigation status. Not addressed. A proper ablation would compare ReLU-SNNK-Adapter against: (1) a standard adapter with identical gating vector and initialization, (2) a standard adapter with matched parameter count (by reducing bottleneck dimension), and (3) a "random projection adapter" that replaces the learned down-projection with a fixed Gaussian matrix but keeps the learned up-projection (to isolate the effect of random projections from the two-tower disentanglement). Without these ablations, the claim that SNNKs specifically drive the performance gains is supported only by the theoretical expressiveness argument (Theorem 3.2, Remark 3.3), not by controlled empirical evidence.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the framing of feedforward layer computation from "fixed primitive to be optimized within its given form" to "kernel-based module whose computational profile can be tuned by varying the number of random features with minimal impact on downstream accuracy." This is not a paradigm shift in the sense of replacing deep learning with kernel methods—the paper's experiments remain firmly within the standard neural network training paradigm, using backpropagation and pretrained Transformers. Rather, it is a reframing of what a feedforward layer is: not a mandatory f(Wx + b) computation with O(ld) cost, but a two-tower kernel evaluation whose parameter count and inference cost can be dialed down dramatically by reducing the random feature dimension m, with the network learning to compensate for the approximation error.

The magnitude of this reframing is best understood as a new axis on the efficiency-expressiveness Pareto frontier that is orthogonal to existing compression methods. Pruning removes weights from an already-trained FFL; distillation trains a smaller student to mimic a larger teacher; quantization reduces numerical precision; low-rank factorization (LoRA, adapters) constrains weight updates to lie in a low-dimensional subspace. SNNKs, in contrast, change the computational primitive itself—replacing the matrix-vector product Wx with a dot-product between separately embedded input and parameter vectors—and the efficiency gain comes from the fact that the embedding dimension m can be chosen independently of the original layer dimensions (l, d). The paper demonstrates concretely that m = 16 random features (in a d = 768 hidden space) suffices for competitive adapter performance on GLUE (Table 1) and that m = 8 enables 40–50% model size reduction in uptrained Transformers (Table 6). These numbers define a new point on the frontier that was previously inaccessible: parameter counts and FLOP counts at 1/3 to 1/2 of standard adapters, with accuracy that matches or exceeds baselines on most tasks.

The paper also reconciles a tension that has existed implicitly in the literature on random features and neural networks. Prior work established that random features can approximate kernels efficiently (Rahimi & Recht, 2007; Choromanski et al., 2018) and that two-layer networks with fixed first-layer weights correspond to kernel machines (Cho & Saul, 2009; Neal, 1996). But these results did not translate into practical compression recipes for deep networks—the kernel approximation view seemed to require m large enough to approximate the kernel accurately, which would negate the computational savings, while the two-layer network view did not extend to deep architectures. The paper's key insight—that in a learning context, m can be far smaller than what kernel approximation theory would demand because the network can adapt to the approximation error—resolves this tension. The evidence is the adapter results: m = 16 in a 768-dimensional space would be a terrible kernel approximator in isolation, yet the SNNK-Adapter outperforms larger standard adapters on 5 of 8 GLUE tasks. This finding changes the research question from "how many random features are needed to approximate a given kernel well?" to "how few random features can we use while still enabling the network to learn effectively?"—a shift from approximation theory to learning dynamics.

Several research directions become more attractive as a result of this work:

• Random-feature-based architectures as a design space. Before this paper, treating the number of random features as a hyperparameter to trade off efficiency against accuracy was not an established practice outside of niche kernel method applications. Now, m becomes a tunable knob analogous to the rank r in LoRA or the bottleneck dimension in adapters—but with a principled connection to the original layer's computation via the kernel formulation, rather than a heuristic low-rank assumption. Researchers can systematically explore this space for different architectures, activations, and tasks.

• Verifier-free fine-tuning via closed-form bundling. The bundled pooler+classifier experiments (Table 5) and the STSB closed-form solution (Spearman 67, computed in seconds on a CPU) are small-scale but provocative. If entire deep networks can be collapsed to y = Wx with fixed nonlinear preprocessing, then for regression tasks, training reduces to solving a linear system—no backpropagation, no learning rate tuning, no GPU required. This is not yet demonstrated at scale (the fully bundled network from Equation 14 is never tested), but the conceptual proof-of-concept shifts this from "impossible" to "unexplored."

• Combining SNNK compression with other efficiency methods. Because SNNKs replace the computational primitive rather than post-processing a trained network, they are orthogonal to pruning, quantization, and distillation. A network could be trained with SNNK layers, then pruned and quantized for additional gains. The paper does not explore this combination, but the orthogonality is a property of the approach, not a contingent empirical finding.

Conversely, some directions become less attractive:

• Developing ever-more-accurate random feature approximations for kernel methods in isolation. The paper's results suggest that in the context of learned neural networks, extremely high-fidelity kernel approximation is unnecessary—the learning process compensates. This redirects research effort away from variance reduction in random feature estimators and toward understanding how approximation error interacts with gradient-based optimization.

• Heuristic adapter design without kernel motivation. Standard adapters (Houlsby et al., 2019) and LoRA (Hu et al., 2022) use low-rank bottlenecks chosen by trial and error. The paper's ReLU-SNNK-Adapter, grounded in the arc-cosine kernel (Theorem 3.2), outperforms standard adapters on most GLUE tasks while using fewer parameters. This suggests that kernel-motivated architecture design may systematically outperform heuristic bottleneck choices, and that exploring the space of kernel-derived architectures (arc-cosine of various orders, Fourier-based SNNKs for different activations, combinations thereof) is more promising than further tuning of ad-hoc low-rank configurations.

Follow-Up Research This Work Enables

Systematic evaluation of Fourier-based URF SNNKs for non-trigonometric activations on real tasks. The paper's theoretical centerpiece—the URF mechanism with four-component Fourier decomposition and softmax kernel linearization (Section 3.1)—is empirically validated only on synthetic pointwise kernel estimation for sine (Figure 2g). The two main practical successes use Cosine-SNNK (simplified two-atom construction) and ReLU-SNNK (arc-cosine construction, independent of URFs). The critical gap: does the full URF mechanism work for ReLU, GELU, tanh, or sigmoid when training real networks? A strong follow-up would implement a ReLU-URF-SNNK (using a smoothed ReLU Fourier transform or distributional treatment, as discussed in Appendix N), a GELU-URF-SNNK, and a tanh-URF-SNNK, then evaluate them head-to-head against the ReLU-SNNK (arc-cosine) and standard adapters on GLUE fine-tuning, matching parameter counts. The key measurement is whether URF-based SNNKs can match or exceed the arc-cosine ReLU-SNNK, and whether the tanh-URF variant succeeds where the brute-force polynomial tanh linearization failed (Table 3: RTE 52.08 vs. baseline 57.5). This would either validate the URF mechanism's practical universality or reveal that only specific kernel constructions (trigonometric, arc-cosine) work well in the context of learned networks, narrowing the paper's claims appropriately.

Training a deep SNNK-only network from scratch on a standard benchmark. Every experiment in the paper where SNNKs perform well modifies a pretrained network: frozen BERT/ViT backbones with SNNK adapters or pooler replacements. The MNIST experiment (Figure 9) replaces only one hidden layer. No experiment tests whether a deep network composed entirely of SNNK layers—no standard FFLs, no pretrained initialization—can be trained from scratch to competitive accuracy. A strong follow-up would take a standard architecture (e.g., a 6-layer MLP or a small Transformer encoder), replace every FFL with ReLU-SNNK or cosine-SNNK layers, and train from scratch on CIFAR-10/100 or a moderate-scale GLUE task, sweeping m to find the efficiency-accuracy frontier. The key measurement is how much larger m must be per layer as depth increases, testing the error propagation analysis (Appendix B) which predicts roughly linear error accumulation with depth and a 1/m variance scaling. If m must grow prohibitively with depth, the "drop-in replacement" claim is valid only for shallow or pretrained networks; if moderate m (e.g., 32–64) suffices for 6–12 layer networks, the claim extends to from-scratch training.

Full-network bundling and closed-form training at scale. The paper's bundling analysis culminates in Equation 14: a fully bundled network reduces to y = Wx where x is composed fixed nonlinear features and W is an unstructured learnable matrix. This is empirically investigated only for bundling the final pooler + classifier layers (Table 5, ~2K parameters), and the closed-form solution is demonstrated only on STSB regression (Spearman 67). A strong follow-up would bundle the entire BERT-base encoder (all 12 Transformer layers) into a single W acting on pre-processed input embeddings, and then train W via closed-form least squares on a regression task (e.g., STSB) or via gradient descent on a classification task (e.g., RTE), comparing against full BERT fine-tuning. The critical measurements: (1) whether the feature map Φ_{f_L} ∘ ... ∘ Φ_{f_1} can be computed tractably (memory and time) for real inputs, (2) the accuracy gap between the fully bundled model and the original BERT, and (3) how m (the random feature dimension at each bundling step) trades off against final accuracy. A negative result—e.g., the composed feature map explodes in dimension or the accuracy collapses—would meaningfully bound the bundling approach's applicability, while a positive result would open the door to backpropagation-free training for regression and rapid linear probing for classification.

Ablation isolating the contribution of the gating vector and initialization in SNNK-Adapters. The ReLU-SNNK-Adapter (Section 4.3.2, Table 1 bottom half) differs from standard adapters in the core SNNK computation AND in architectural details: a learned gating vector v initialized to zero, and the specific random projection pattern. The paper acknowledges that "other detailed initialization schemes are not studied" (Appendix D) and that zero-initialization of v is critical for stability. A strong follow-up would run a clean ablation: (1) ReLU-SNNK-Adapter with vs. without the gating vector v (using alternative identity-initialization strategies); (2) standard adapter with an identical gating vector added; (3) "random projection adapter" where the learned down-projection matrix is replaced by a fixed Gaussian projection (but keeping the learned up-projection and activation), to isolate the effect of random projections from the two-tower dot-product structure. The key measurement is which component drives the performance gain: the two-tower kernel formulation, the random projection pattern, the gating mechanism, or an interaction. The paper attributes the gains to SNNK expressiveness (Theorem 3.2, Remark 3.3), but without this ablation, alternative explanations (gating, optimization dynamics) cannot be ruled out.

SNNK-based adapters combined with other PEFT methods. The paper notes that SNNKs "are completely orthogonal to techniques such as gating mechanism in (Mao et al., 2022) or algorithms relying on dropping suitable adapter layers (Moosavi et al., 2022; Rücklé et al., 2021)" and "can be easily combined with them" (Section 4.3.2). However, no combination experiment is run. A strong follow-up would combine ReLU-SNNK-Adapter with LoRA (Hu et al., 2022)—applying LoRA's low-rank updates to the SNNK's learned parameter matrix A, or applying SNNK adapters alongside LoRA updates to attention weights—and with adapter-dropping regularization (Rücklé et al., 2021). The key measurement is whether the efficiency gains compound: can SNNK + LoRA achieve higher accuracy than either alone at a fixed total parameter budget, or achieve the same accuracy with fewer total parameters? The orthogonality claim is plausible but untested; a negative result (the methods interfere) would be equally informative for practitioners designing multi-method compression pipelines.

Practical Applications and Downstream Use Cases

Parameter-efficient fine-tuning of large Transformer models for multi-task serving. In production systems where a single pretrained large language model (e.g., BERT, RoBERTa, or a decoder-only model) must be fine-tuned for many downstream tasks and served concurrently, storing a full copy of the fine-tuned model per task is prohibitively expensive. SNNK-Adapters with m = 16 random features reduce the trainable parameters per task from ~0.9M (standard adapter) or ~1.8M (Houlsby adapter) to ~0.3M (Table 8). For a BERT-base model (110M parameters) serving 100 downstream tasks, standard adapters add ~90M total task-specific parameters; SNNK-Adapters add ~30M. The storage savings (~3×) compound with the number of tasks. The competitive or superior accuracy on 5 of 8 GLUE tasks (Table 1) means the compression does not come at a significant accuracy cost for most tasks. The gating vector initialization (v = 0) ensures the adapted model starts as the pretrained model, which is critical for production stability.

On-device deployment of vision Transformers with reduced memory footprint. The uptraining experiments (Appendix J.4, Table 6) show that bundling the top 6 FFN blocks of ViT-base reduces model size from 346 MB to 176.42 MB (~49% reduction) and inference FLOPs from 563B to 384B (~32% reduction) with "minimal impact on accuracy" (Figure 13). For edge deployment scenarios—smartphone camera apps, drones, IoT devices—where model size is constrained by device storage and inference latency is constrained by battery and thermal budgets, a 49% smaller model with 32% faster inference is a substantial practical improvement. Unlike quantization (which may require specialized hardware support for INT8/FP16 inference) or pruning (which may produce irregular sparsity patterns that are inefficient on standard hardware), SNNK compression produces a dense, regular matrix W (Equation 14) that runs efficiently on any standard GPU or CPU. The fact that the bundling can be applied selectively (e.g., only to the top layers, keeping early layers in their original form to preserve low-level feature extraction) gives practitioners fine-grained control over the accuracy-efficiency tradeoff.

Rapid task adaptation via closed-form linear regression on fixed features. The bundled pooler+classifier experiment (Appendix J.3, Table 5) and the STSB closed-form demonstration (Spearman 67, computed in "barely a few seconds on a CPU") point to a deployment pattern for regression and linear classification tasks: precompute the fixed feature vector x for all inputs using the composed SNNK feature maps, then solve for the optimal linear weights W via least squares or logistic regression. This completely bypasses gradient-based fine-tuning—no GPU, no hyperparameter tuning, no convergence monitoring. The practical benefit is speed and simplicity: for a new regression task (e.g., predicting user engagement scores from text), one could take a frozen pretrained Transformer, pass all training examples through the SNNK-bundled feature extractor (a forward pass, parallelizable across examples), and solve the linear system in seconds on a CPU. The accuracy is competitive but not state-of-the-art (STSB 67 vs. 78.1 for full pooler+classifier training), so this is appropriate for scenarios where rapid deployment and simplicity outweigh marginal accuracy gains—rapid prototyping, resource-constrained settings without GPU access, or tasks where the pretrained representations are already highly informative.

When to Prefer This Method

The paper positions SNNKs against several named alternatives, though not always with explicit tradeoff articulation. The following decision rules are grounded in the paper's empirical comparisons:

• Prefer SNNK-Adapters over standard adapters (Houlsby/Pfeiffer) when the total parameter budget for task-specific fine-tuning is tightly constrained (e.g., multi-task serving with 100+ tasks, on-device storage limits) AND the downstream tasks are of small-to-moderate size (RTE, MRPC, CoLA, SST-2, STSB) where the ReLU-SNNK-Adapter outperforms standard adapters at 1/3 the parameter count (Table 1, Table 8). On large datasets (MNLI, QQP, QNLI), standard adapters or LoRA may still hold an advantage (Table 8: LoRA achieves 84.1 on MNLI vs. 82.06 for SNNK-Adapter), though the paper notes that increasing m closes this gap (Figure 5).

• Prefer bundling via SNNK uptraining over standard model compression (pruning/quantization) when the deployment hardware is standard (no sparse matrix support, no INT8 acceleration) and the model architecture includes large FFN blocks that dominate parameter count—the bundling produces a dense compressed matrix that runs efficiently on any hardware, unlike pruning's irregular sparsity. The 40–50% size reduction and 32% FLOP reduction (Table 6) are achieved with minimal accuracy loss (Figures 12–13), making this suitable for BERT and ViT models in throughput-constrained or memory-constrained serving. However, for latency-critical applications where the serial dependency of bundling (each layer's random features must be computed before the next) matters, the wall-clock speedup may be less than the FLOP reduction suggests.

• Prefer the closed-form bundled solution over gradient-based fine-tuning when the task is regression (mean squared error loss) or binary/multiclass classification with few training examples, the pretrained backbone provides high-quality representations (e.g., BERT on STSB), and rapid deployment without GPU infrastructure is prioritized over maximizing accuracy. The STSB demonstration (Spearman 67 in seconds on CPU, Appendix J.3) shows viability; the gap to the trained baseline (78.1) defines the accuracy cost of avoiding backpropagation.

• Prefer standard FFLs or LoRA over SNNKs when training deep networks from scratch with no pretrained initialization—the paper provides no evidence that SNNKs can learn hierarchical representations from random initialization across many layers, and the error propagation analysis (Appendix B) suggests that variance accumulates with depth. Until from-scratch training of deep SNNK architectures is empirically demonstrated, pretrained backbones remain a prerequisite for the approach.