KAN: Kolmogorov-Arnold Networks

ArXiv: 2404.19756

🎯 Pitch

Replace every weight in a neural network with a learnable spline and you get a KAN, which scales with a remarkable exponent of ~4 on PDEs while MLPs plateau, and can even be symbolically pruned to rediscover physical laws like knot invariants.

1. Executive Summary

This paper introduces Kolmogorov–Arnold Networks (KANs) as alternatives to Multi-Layer Perceptrons (MLPs), replacing fixed activation functions on nodes with learnable activation functions parametrized as splines on edges. Evaluated on synthetic regression tasks, special functions, Feynman physics equations, and PDE solving with small-scale models, KANs demonstrate faster neural scaling laws than MLPs—achieving a scaling exponent of approximately 4 empirically compared to MLPs' slower, plateauing behavior—and superior accuracy with dramatically fewer parameters (e.g., a 200-parameter KAN reaching 81.6% test accuracy versus a ~3×10⁵-parameter MLP at 78.0% on knot-theoretic signature prediction). Through sparsification, pruning, and symbolification techniques, KANs further enable interpretable scientific discovery by extracting symbolic formulas and revealing compositional structure, establishing that KANs outperform MLPs in accuracy and interpretability on small-scale AI + Science tasks while remaining 10× slower to train.

2. Context and Motivation

The Core Problem: MLPs Are the Default, Not the Optimal, Building Block for Function Approximation

The fundamental question this paper tackles is whether the Multi-Layer Perceptron—the workhorse of deep learning for decades—is actually the best architecture for approximating nonlinear functions, or whether we have settled into a local optimum out of historical momentum. This is not a question about training techniques or scaling strategies applied to MLPs; it's a question about the architectural primitive itself: the placement of fixed, non-learnable activation functions on nodes, separated from linear weight matrices.

The authors identify specific, concrete deficiencies in MLPs that motivate searching for an alternative:

MLPs treat linear transformations and nonlinearities as separable, sequential operations. As formalized in Equation (2.9), an MLP layer consists of an affine transformation $W$ followed by a fixed nonlinearity $\sigma$ : $\text{MLP}(x) = (W_{L-1} \circ \sigma \circ W_{L-2} \circ \sigma \circ \cdots \circ W_1 \circ \sigma \circ W_0)x$ . This decomposition forces the model to learn compositional structure through the linear weights alone, while the nonlinearities remain passive, unchanging across the entire input space. In high dimensions, this means the model must represent complex multivariate interactions purely via linear combinations, relying on depth to build up nonlinear expressivity layer by layer. The question is whether this separation is a necessary evil or an arbitrary constraint.

MLPs lack a notion of "fine-graining" a function approximation. Once an MLP is trained, there is no principled mechanism to improve its approximation quality by, say, increasing its resolution in targeted regions of the input space. To get better accuracy, one must train an entirely new, larger model from scratch—what the paper calls "neural scaling laws" that are "slow" and "expensive to obtain" (Section 2.4). This is fundamentally at odds with how numerical analysis approaches function approximation, where methods like splines, finite elements, or spectral methods can refine their approximation by adding more basis functions adaptively, without re-solving the entire problem.

MLPs are notoriously uninterpretable. In transformers, the paper notes, "MLPs consume almost all non-embedding parameters and are typically less interpretable (relative to attention layers) without post-analysis tools" (Section 1). Even with sophisticated mechanistic interpretability techniques—saliency maps, sparse autoencoders, circuit analysis—understanding what an MLP has learned about a function requires substantial post-hoc effort. The weights themselves are opaque matrices of numbers; the fixed activation functions reveal nothing about the structure of the target function. This is particularly frustrating in scientific applications, where the goal is not just prediction accuracy but understanding the underlying functional relationship.

MLPs suffer from catastrophic forgetting in continual learning (Section 3.5). Because MLPs use global activation functions (ReLU, tanh, SiLU), updating the network on new data in one region of the input space can propagate changes unpredictably across the entire function approximation, overwriting previously learned information. The authors argue this is a fundamental architectural limitation: "Most artificial neural networks, including MLPs, do not have this notion of locality, which is probably the reason for catastrophic forgetting" (Section 3.5).

Why This Problem Matters

The significance of finding an MLP alternative extends across multiple scales of AI practice:

For AI + Science, interpretability is functional, not cosmetic. When a physicist wants to extract a mobility edge from condensed matter data (Section 4.4) or a mathematician wants to discover relations among knot invariants (Section 4.3), they need more than a black-box predictor. They need to read off the symbolic structure of the learned function—to see that signature depends on $\lambda\mu_r/(\mu_r^2 + \mu_i^2)$ rather than just getting 81.6% accuracy. The paper positions KANs as a "language model for AI + Science" (Section 6) precisely because the scientific language is functions, and KANs are "composed of interpretable functions, so when a human user stares at a KAN, it is like communicating with it using the language of functions."

For resource-constrained deployment, parameter efficiency matters. The paper demonstrates that a KAN with ~200 parameters can match or exceed a 3×10⁵-parameter MLP on the knot signature prediction task (Table 3). This is not a marginal improvement—it is three orders of magnitude fewer parameters for better accuracy. For on-device deployment, edge computing, or any scenario where model size is the binding constraint, this efficiency gap is transformative.

For continual learning, locality is essential. Catastrophic forgetting remains a major obstacle to building AI systems that can accumulate knowledge over time. The paper's demonstration that KANs can learn sequential Gaussian peaks without overwriting previous knowledge (Figure 3.4) suggests an architectural solution rather than an algorithmic workaround. This matters for any application where data arrives sequentially and the model must retain previously acquired capabilities.

For the theory of neural scaling laws, a new bound opens possibilities. The paper's approximation theorem (Theorem 2.1) predicts a scaling exponent $\alpha = k+1 = 4$ for cubic B-splines, independent of input dimension under the assumption of a smooth Kolmogorov-Arnold representation. This contrasts with the standard curse-of-dimensionality bound $\alpha = (k+1)/d$ from Sharma & Kaplan. If such representations exist for real-world functions (a big "if"), KANs could theoretically scale far more efficiently than MLPs to high-dimensional problems.

Prior Approaches and Where They Fall Short

The paper identifies several strands of prior work, each with specific limitations that motivate the KAN approach:

The Kolmogorov-Arnold representation theorem has been explored before, but shallowly. As the paper acknowledges, "the possibility of using Kolmogorov-Arnold representation theorem to build neural networks has been studied" in prior work spanning decades (Section 1). However, existing attempts share critical limitations:

They stick to the original 2-layer, width-(2n+1) form of the theorem (Equation 2.1). This shallow architecture has limited expressive power, especially when the constituent 1D functions are required to be smooth splines. The paper explicitly notes that "such a network is known to be too simple to approximate any function arbitrarily well in practice with smooth splines!" (Section 2.2).
They predate modern training infrastructure. Much of this prior work existed "even predating back-propagation" (Section 5), meaning it could not leverage the optimization techniques that made deep learning successful. The networks were small, the training was primitive, and the empirical demonstrations were "rather limited or artificial toy experiments."
The pathological behavior of the theorem's inner functions created pessimism. Girosi and Poggio [20] famously argued that the Kolmogorov-Arnold theorem is "irrelevant" for machine learning because the univariate functions in the representation theorem can be non-smooth "and even fractal" (Section 2.1). This "sentenced [the theorem] to death in machine learning, regarded as theoretically sound but practically useless." The paper's key reframing is that "most functions in science and daily life are often smooth and have sparse compositional structures, potentially facilitating smooth Kolmogorov-Arnold representations" (Section 2.1)—essentially arguing that the pathological worst case doesn't matter if typical functions in practice admit smooth representations.

Splines are excellent for low-dimensional function approximation but suffer from the curse of dimensionality. Spline-based methods can approximate 1D functions with high accuracy and provide local control (adjusting coefficients in one region doesn't affect distant regions). However, splines cannot exploit compositional structure—trying to approximate a high-dimensional function directly with a multivariate spline requires a number of basis functions exponential in the dimension. As the paper puts it: "splines have a serious curse of dimensionality (COD) problem, because of their inability to exploit compositional structures" (Section 1).

MLPs exploit compositional structure but are inefficient at learning univariate functions. MLPs, through their layered architecture, can learn hierarchical compositions—this is why they work at all in high dimensions. But because their nonlinearities are fixed, they are "less accurate than splines in low dimensions, because of their inability to optimize univariate functions" (Section 1). The example in Equation (1.1) crystallizes this: to learn $f(x) = \exp(\frac{1}{N}\sum_i \sin^2(x_i))$ , an MLP must approximate $\exp$ and $\sin$ using compositions of ReLU activations, which requires many parameters and yields mediocre approximations. A spline could nail the exponential and sine perfectly, but would choke on the 100-dimensional input space.

Symbolic regression is brittle and opaque as an alternative for scientific discovery. One might argue that symbolic regression (SR) methods—genetic algorithms, Eureka, PySR—already solve the interpretability problem. The paper identifies specific failures of SR relative to KANs (Section 2.5.2 and Section 4.4):

SR is all-or-nothing. "They either return a success or a failure in the end without outputting interpretable intermediate results." If SR fails to find the formula, the user has no recourse—no gradient of understanding, no partial insight.
SR lacks user interactivity. "The way we visualize KANs is like displaying KANs' 'brain' to users, and users can perform 'surgery' (debugging) on KANs. This level of control is typically unavailable for SR."
SR cannot handle special functions not in its library. If the target function contains a Bessel function and Bessel isn't in the symbol library, SR is helpless. KANs can "use splines to approximate it numerically anyway" (Section 2.5.2), and the user can recognize the wiggly shape from the visualization and manually set the activation to J_0.
SR requires specifying a symbol library upfront, which presupposes domain knowledge about what functions might appear. KANs allow the user to first see the learned activations and then decide on symbolic forms.

Physics-Informed Neural Networks (PINNs) use MLPs and inherit their limitations. The paper demonstrates (Section 3.4) that for solving a Poisson equation via PINNs, a 2-layer width-10 KAN achieves 100× better accuracy (10⁻⁷ vs. 10⁻⁵ MSE) with 100× fewer parameters (10² vs. 10⁴) than a 4-layer width-100 MLP. The implication is that the MLP backbone is the bottleneck in PINN performance, not the physics-informed loss formulation itself.

Mechanistic interpretability treats symptoms, not causes. Most mechanistic interpretability research is "passive," "focusing on understanding existing neural networks trained with standard methods" (Section 5). This requires post-hoc analysis tools and struggles with the inherent opacity of weight matrices. The paper positions KANs in the "active" MI category, where "the model and training method are by design interpretable"—architectural interpretability rather than retroactive explanation.

How This Paper Positions Itself

The paper's positioning can be understood as a three-part thesis:

1. The Kolmogorov-Arnold theorem deserves revitalization, not burial. The paper argues that the theorem's dismissal was premature because it considered only the shallow, fixed-width form and the worst-case behavior of inner functions. By generalizing to arbitrary widths and depths—stacking KAN layers just as MLPs stack fully-connected layers—the architecture gains the expressive power needed for practical function approximation. The insight is that "deeper representations may bring the advantages of smoother activations" (Section 2.3), effectively trading depth for smoothness: a function that requires non-smooth 1D components in a 2-layer representation might admit smooth 1D components in a 3-layer representation. The example in Equation (2.13) makes this concrete: a 4-variable function that "may not admit a 2-Layer KAN with smooth activations" can be "smoothly represented by a [4, 2, 1, 1] KAN which is 3-Layer."

2. KANs are a hybrid that combines the strengths of splines and MLPs while avoiding their weaknesses. This is the paper's central architectural claim: "KANs are nothing more than combinations of splines and MLPs, leveraging their respective strengths and avoiding their respective weaknesses" (Section 1). Specifically:

From MLPs, KANs inherit the ability to learn compositional structure ("external degrees of freedom" in the paper's terminology) through the connectivity of the computation graph across multiple layers. This is what allows them to handle high-dimensional functions without suffering the curse of dimensionality.
From splines, KANs inherit the ability to accurately approximate univariate functions ("internal degrees of freedom") through learnable B-spline coefficients. This is what allows them to nail exponentials, sines, and Bessel functions that MLPs struggle with.

The paper introduces the terminology external vs. internal degrees of freedom specifically to articulate this hybrid nature (Section 2.4): "External dofs (that MLPs also have but splines do not) are responsible for learning compositional structures of multiple variables. Internal dofs (that splines also have but MLPs do not) are responsible for learning univariate functions."

3. The combination of accuracy and interpretability makes KANs uniquely suited for AI + Science as a "collaborator," not just a predictor. The paper's most ambitious framing is that KANs enable a new paradigm of human-AI collaboration in scientific discovery. The key idea is that KANs speak the native language of science—functions—so scientists can literally read what the model has learned by inspecting the learned activation functions. This is operationalized through the simplification pipeline described in Section 2.5: sparsification → pruning → symbolification → formula extraction. The scientist doesn't just get a prediction; they get a visualization of the learned compositional structure and can interact with it: "users have more control over KANs as compared to SR due to KANs' transparency. The way we visualize KANs is like displaying KANs' 'brain' to users, and users can perform 'surgery' (debugging) on KANs."

This collaboration paradigm is explicitly contrasted with the "AI scientist" approach: "The intrinsic difficulty of (fully automated) AI scientists is that it is hard to make human preferences quantitative... it is more desirable for scientists to have an AI that can speak the scientific language (functions) and can conveniently interact with inductive biases of individual scientist(s) to adapt to a specific scientific domain" (Section 6). The paper envisions KANs as the first step toward AI that helps scientists rather than replacing them.

4. The neural scaling law analysis reframes what "good scaling" means. By deriving a scaling exponent $\alpha = k+1 = 4$ (independent of dimension, under smooth KA representation assumptions) and showing empirically that KANs approach this bound while MLPs plateau, the paper makes a theoretical claim about the fundamental efficiency of its architecture. This is not just "KANs are better"—it's "KANs have a provably faster asymptotic scaling rate than MLPs when the target function admits a smooth KA representation." The paper connects this to the broader neural scaling laws literature (Sharma & Kaplan, Michaud et al., Poggio et al.) and positions its $\alpha = 4$ bound as "the largest and best scaling exponent compared to other works" (Section 2.3).

The paper is careful, however, to scope its claims. The decision tree in Figure 6.1 explicitly acknowledges practical limitations: KANs are "usually 10x slower than MLPs, given the same number of parameters" and the authors "did not try hard to optimize KANs' efficiency." The recommendation is pragmatic: use MLPs when training speed matters; consider KANs when "you care about interpretability and/or accuracy, and slow training is not a major concern." This honesty about trade-offs, combined with the ambitious theoretical and empirical claims, creates a positioning that is both transformative in vision and grounded in current limitations.

3. Technical Approach

3.1 Reader Orientation

This paper is fundamentally an architectural innovation paper that proposes a new neural network primitive—the Kolmogorov-Arnold Network (KAN)—as a drop-in replacement for Multi-Layer Perceptrons. Rather than learning linear weights between neurons with fixed activation functions applied at nodes, KANs learn univariate functions (parametrized as B-splines) on every edge, with nodes performing only summation. The core idea is that this architectural change unifies the strengths of MLPs (compositional structure learning) and splines (accurate univariate function approximation) into a single model class that is simultaneously more parameter-efficient, more interpretable, and capable of faster neural scaling than MLPs for function approximation tasks.

The system solves the problem of function approximation with simultaneous accuracy and interpretability: given input-output pairs from an unknown multivariate function, it learns not just to predict outputs accurately, but to reveal the compositional structure of that function in a form that a human scientist can read off directly—identifying which variables interact, what univariate transformations are applied, and ultimately extracting symbolic formulas.

3.2 Big-Picture Architecture (Diagram in Words)

A KAN is a stack of KAN layers, each of which maps an input vector to an output vector through a matrix of learnable 1D functions rather than through a linear weight matrix followed by a fixed nonlinearity. The major components are:

KAN Layer (Φ): A matrix of univariate functions $\phi_{j,i}$ connecting each input node $i$ to each output node $j$ . Each $\phi_{j,i}$ is a spline-plus-basis function that transforms a single scalar input to a single scalar output. There are no linear weight matrices anywhere in the network.
B-Spline Parametrization: Each edge function $\phi(x)$ is the sum of a fixed basis function (SiLU, $x/(1+e^{-x})$ ) and a learnable B-spline function (a linear combination of local cubic B-spline basis functions on a grid). This gives the function both global structure (from the basis) and local adaptivity (from the B-spline coefficients).
Grid Extension Mechanism: The grid of B-spline knots can be refined during training—starting coarse (e.g., 3 intervals) and progressively increasing to finer grids (up to 1000 intervals)—without reinitializing the network. This provides a natural "fine-graining" capability absent from MLPs.
Simplification Pipeline: A post-training workflow consisting of sparsification (L1 + entropy regularization on activation function magnitudes), pruning (removing nodes whose incoming and outgoing activation scores fall below a threshold), symbolification (manually or automatically replacing learned spline activations with known symbolic forms like sin, exp, x^2), and formula extraction (computing the symbolic expression of the output node via SymPy).
Training Objective: Standard supervised regression or classification loss (MSE or cross-entropy) augmented with sparsity-inducing regularization on the L1 norms and entropy of each layer's activation functions.

Information flows as follows: an input vector $x_0$ enters → each component $x_{0,i}$ flows through the $j$ th edge function $\phi_{0,j,i}$ to produce $\tilde{x}_{0,j,i} = \phi_{0,j,i}(x_{0,i})$ → at the next layer's $j$ th node, all incoming post-activations are summed to produce $x_{1,j} = \sum_i \tilde{x}_{0,j,i}$ → this summed value flows through the next layer's edge functions → repeat for $L$ layers → final output $x_L$ is the KAN's prediction.

3.3 Roadmap for the Deep Dive

First, the Kolmogorov-Arnold representation theorem and its generalization to deep KANs, because this is the mathematical foundation that motivates the entire architecture and distinguishes KANs from prior attempts to use the theorem.
Second, the KAN layer formal definition, including the notation for activation functions, the summation at nodes, and the compositional structure of a deep KAN—since this establishes the computational framework.
Third, the B-spline parametrization of edge functions, including the residual basis-plus-spline decomposition, the initialization strategy, and the grid update mechanism—because the parametrization determines what KANs can learn and how they train.
Fourth, the grid extension technique for progressive accuracy improvement, along with the least-squares initialization procedure that transfers learned coefficients from coarse grids to fine grids—since this is a unique capability that enables the observed scaling laws.
Fifth, the theoretical approximation guarantee (Theorem 2.1), which predicts the $\alpha = k+1$ scaling exponent and provides the formal justification for why KANs can beat the curse of dimensionality under smooth KA representation assumptions.
Sixth, the simplification pipeline (sparsification, pruning, symbolification) that makes KANs interpretable, since interpretability is the second major claimed advantage beyond accuracy.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an architectural innovation paper whose core idea is that replacing the fixed-activation-on-nodes, linear-weight-on-edges paradigm of MLPs with a learnable-activation-on-edges, summation-on-nodes paradigm creates a model class that inherits the compositional expressivity of deep networks and the univariate approximation accuracy of splines, while also enabling natural interpretability through visualization and symbolic regression of the learned 1D functions.

The Kolmogorov-Arnold Representation Theorem and Its Generalization

The foundation of the KAN architecture is the Kolmogorov-Arnold representation theorem, which states that any multivariate continuous function $f: [0, 1]^n \to \mathbb{R}$ can be expressed as a finite composition of univariate continuous functions and addition. The exact statement (Equation 2.1) is:

$f(x) = f(x_1, \cdots, x_n) = \sum_{q=1}^{2n+1} \Phi_q\left(\sum_{p=1}^n \phi_{q,p}(x_p)\right)$

where $\phi_{q,p}: [0, 1] \to \mathbb{R}$ are the "inner" univariate functions (one for each input dimension $p$ and each term $q$ in the sum, totaling $n \times (2n+1)$ functions), and $\Phi_q: \mathbb{R} \to \mathbb{R}$ are the "outer" univariate functions (one for each term $q$ , totaling $2n+1$ functions). The $x_p$ are the input variables, and the output is a scalar.

What it computes: The formula says that to evaluate $f$ at any point $x$ , you: (1) apply $n \times (2n+1)$ different univariate functions $\phi_{q,p}$ to each input coordinate $x_p$ individually, (2) sum the results for each $q$ across all input dimensions to get $2n+1$ intermediate scalar values, (3) apply the $2n+1$ outer functions $\Phi_q$ to these scalar sums, and (4) sum those $2n+1$ outer results to get the final output. Every operation is either a 1D function evaluation or addition—there is no genuine multivariate operation besides summation.

Why this form: The theorem is profound because it decomposes an arbitrary multivariate function—which might seem to require genuinely multivariate operations—into only univariate functions and addition. In a sense, "the only true multivariate function is addition, since every other function can be written using univariate functions and sum" (Section 2.1). For machine learning, this is potentially revolutionary: it suggests that learning a high-dimensional function could reduce to learning a polynomial number of 1D functions, completely bypassing the curse of dimensionality if you can find the right univariate components.

The historical pessimism and why it was wrong. The paper explicitly acknowledges that the theorem was "basically sentenced to death in machine learning" (Section 2.1) because the inner functions $\phi_{q,p}$ can be "non-smooth and even fractal, so they may not be learnable in practice." The paper pivots on two counterarguments: (1) Most functions encountered in science and daily life are smooth and have sparse compositional structures, meaning they likely admit smooth Kolmogorov-Arnold representations even if worst-case functions don't. (2) We need not stick to the original formula with exactly $2n+1$ hidden units and exactly two layers—we can generalize to arbitrary depths and widths, and "deeper representations may bring the advantages of smoother activations" (Section 2.3). The insight is that a function requiring pathological 1D components in a shallow representation might admit perfectly smooth 1D components in a deeper representation, because depth allows the model to decompose the function more gradually.

Generalizing to deep KANs. The original theorem corresponds to a 2-layer KAN with shape $[n, 2n+1, 1]$ . The paper's key architectural move is to notice that this is simply a composition of two KAN layers—an inner layer with $n$ inputs and $2n+1$ outputs, and an outer layer with $2n+1$ inputs and $1$ output—and that nothing prevents stacking more such layers. The paper explicitly states that "to the best of our knowledge, there is not yet a 'generalized' version of the theorem that corresponds to deeper KANs," so this generalization is an empirical hypothesis (supported by the experiments showing deeper KANs outperform shallower ones, e.g., the [4, 2, 2, 1] KAN vs. [4, 9, 1] KAN in Figure 3.1 bottom-right).

KAN Layer Formal Definition

A KAN layer with $n_{\text{in}}$ inputs and $n_{\text{out}}$ outputs is defined as a matrix of 1D functions (Equation 2.2):

$\Phi = \{\phi_{q,p}\}, \quad p = 1, 2, \cdots, n_{\text{in}}, \quad q = 1, 2, \cdots, n_{\text{out}}$

where $\phi_{q,p}: \mathbb{R} \to \mathbb{R}$ is the univariate activation function connecting input node $p$ to output node $q$ , and each $\phi_{q,p}$ has its own set of trainable parameters (the B-spline coefficients and the scale factors, described below). The index $q$ runs over output nodes and index $p$ runs over input nodes.

What it computes (forward pass): Given an input vector $x_l \in \mathbb{R}^{n_l}$ at layer $l$ , the $j$ th component of the output vector $x_{l+1} \in \mathbb{R}^{n_{l+1}}$ is computed as (Equation 2.5):

$x_{l+1, j} = \sum_{i=1}^{n_l} \phi_{l, j, i}(x_{l,i})$

where $\phi_{l,j,i}$ is the activation function on the edge from node $(l,i)$ to node $(l+1,j)$ , and $x_{l,i}$ is the activation value at node $(l,i)$ . Each input component flows independently through its own dedicated 1D function to each output node, and the output node simply sums all its incoming transformed values—there is no activation function applied at the node itself.

Why this form: This is the critical departure from MLPs. In an MLP layer, you compute $x_{l+1} = \sigma(W_l x_l)$ : a linear combination $W_l x_l$ that mixes all input components together, followed by a fixed pointwise nonlinearity $\sigma$ . In a KAN layer, you compute $x_{l+1,j} = \sum_i \phi_{l,j,i}(x_{l,i})$ : each $(i,j)$ pair gets its own personalized nonlinear transformation $\phi_{l,j,i}$ applied before any mixing occurs, and the only mixing is a simple sum. The learnable nonlinearity and the "weight" are fused into a single entity—the edge function—rather than being separated into $W$ and $\sigma$ .

Matrix form (Equation 2.6): The full layer can be written as:

$x_{l+1} = \begin{pmatrix} \phi_{l,1,1}(\cdot) & \phi_{l,1,2}(\cdot) & \cdots & \phi_{l,1,n_l}(\cdot) \\ \phi_{l,2,1}(\cdot) & \phi_{l,2,2}(\cdot) & \cdots & \phi_{l,2,n_l}(\cdot) \\ \vdots & \vdots & & \vdots \\ \phi_{l,n_{l+1},1}(\cdot) & \phi_{l,n_{l+1},2}(\cdot) & \cdots & \phi_{l,n_{l+1},n_l}(\cdot) \end{pmatrix} x_l$

where the "function matrix" $\Phi_l$ acts on $x_l$ such that each element $\Phi_l[j,i]$ is a function $\phi_{l,j,i}(\cdot)$ applied to the $i$ th component of $x_l$ , and the results are summed row-wise (across $i$ ). This is notationally analogous to $x_{l+1} = W_l x_l$ in linear algebra, but every scalar weight $w_{ji}$ is replaced by a full function $\phi_{l,j,i}$ .

Full KAN (Equation 2.7): A deep KAN with $L$ layers is the composition:

$\text{KAN}(x) = (\Phi_{L-1} \circ \Phi_{L-2} \circ \cdots \circ \Phi_1 \circ \Phi_0)(x)$

where each $\Phi_l$ is a KAN layer as defined above. The shape is specified by the integer array $[n_0, n_1, \cdots, n_L]$ , where $n_0$ is the input dimension, $n_L$ is the output dimension (typically 1 for scalar regression), and the intermediate $n_l$ are the widths of hidden layers. The total number of activation functions in the network is $\sum_{l=0}^{L-1} n_l n_{l+1}$ .

Contrast with MLP (Equation 2.9): An MLP of the same depth is:

$\text{MLP}(x) = (W_{L-1} \circ \sigma \circ W_{L-2} \circ \sigma \circ \cdots \circ W_1 \circ \sigma \circ W_0)(x)$

The structural difference is that MLPs interleave affine maps $W_l$ with fixed nonlinearities $\sigma$ , while KANs have the nonlinearities baked into the "weight" functions $\Phi_l$ themselves. All operations in both architectures are differentiable, so KANs are trained with standard backpropagation—the spline parametrization is fully differentiable with respect to its coefficients.

The depth-width tradeoff visualization. Figure 0.1 (c) and (d) illustrate this: a 3-layer MLP shows linear weight matrices $W_1, W_2, W_3$ interleaved with fixed $\sigma_1, \sigma_2$ , while a 3-layer KAN shows function matrices $\Phi_1, \Phi_2, \Phi_3$ where each edge carries a learnable curve. The nodes in the MLP apply $\sigma$ ; the nodes in the KAN only sum.

B-Spline Parametrization of Edge Functions

Each edge function $\phi(x)$ is parametrized as a sum of two components (Equations 2.10–2.12):

$\phi(x) = w_b \cdot b(x) + w_s \cdot \text{spline}(x)$

where $b(x) = \text{silu}(x) = x/(1 + e^{-x})$ is a fixed basis function (the SiLU/Swish activation), $\text{spline}(x) = \sum_i c_i B_i(x)$ is a learnable linear combination of B-spline basis functions, and $w_b, w_s$ are scalar weight factors (both trainable, initialized to $w_s = 1$ and $w_b$ via Xavier initialization).

What $b(x)$ provides: The SiLU basis function gives the activation function a non-zero "backbone" that is globally defined (non-zero everywhere) and smooth. This ensures that even before the spline component has learned anything useful, the edge function is not degenerate—it's at least a SiLU-shaped curve, which is a reasonable default activation function. This is analogous to residual connections in standard networks: the basis function provides a "skip connection" that allows gradients to flow even when the spline component is near zero.

What $\text{spline}(x)$ provides: The B-spline component is the learnable, locally adaptive part. B-splines are piecewise polynomial functions defined on a grid of knot points. For a grid with $G$ intervals (and $G+1$ grid points) on a bounded domain $[a, b]$ , and spline order $k$ (typically $k=3$ for cubic splines), there are $G+k$ B-spline basis functions. The $i$ th B-spline $B_i(x)$ is non-zero only on $[t_{-k+i}, t_{i+1}]$ , where the $t_j$ are the augmented knot sequence $\{t_{-k}, \ldots, t_{-1}, t_0, \ldots, t_G, t_{G+1}, \ldots, t_{G+k}\}$ . The coefficients $c_i$ are the trainable parameters that determine the shape of the spline within its support.

Why this decomposition: The basis-plus-spline decomposition separates global structure (captured by the SiLU backbone) from local adjustments (captured by the B-spline). This is important because B-splines are defined on bounded intervals—spline coefficients learned on one region of the input space don't affect the function outside that region. The SiLU basis ensures the function is well-defined and reasonable everywhere, while the spline fine-tunes it locally. The scale factors $w_b$ and $w_s$ control the relative contribution of each component, and they are kept trainable (rather than absorbed into $b$ and $\text{spline}$ ) to "better control the overall magnitude of the activation function" (Section 2.2).

Initialization strategy. At initialization, the spline component is set to be approximately zero everywhere by drawing $c_i \sim \mathcal{N}(0, \sigma^2)$ with a small $\sigma = 0.1$ . This means that initially, $\phi(x) \approx w_b \cdot \text{silu}(x)$ , i.e., the KAN starts behaving similarly to a network with SiLU activations (since the spline component is negligible). The weight $w_b$ is initialized via Xavier initialization (the same scheme used for linear layers in MLPs), and $w_s = 1$ . This initialization ensures that training starts from a reasonable baseline and the spline component is gradually "activated" as training proceeds.

Grid update mechanism. B-splines are defined on a fixed bounded grid, but during training, the input activations can drift outside the initial grid boundaries (e.g., if the grid was set to $[-1, 1]$ but activations grow to $[-2, 2]$ ). To handle this, the paper updates the grid for each activation function on the fly based on the observed range of its input values. The grid endpoints are set to cover the range of activations seen so far, and the B-spline basis functions are redefined on the updated grid. The paper mentions that alternative approaches were tried (e.g., using normalization to fix the input range) but "its performance is inferior to our current approach."

Parameter count. For a KAN with depth $L$ , equal width $N$ at all layers, spline order $k$ (usually $3$ ), and $G$ grid intervals: each of the $N^2(L-1)$ edge functions has $G+k$ spline coefficients, plus the two scale parameters $w_b, w_s$ . So total parameters are approximately $O(N^2 L (G+k)) \sim O(N^2 L G)$ . For comparison, an MLP with the same depth and width has $O(N^2 L)$ parameters (just the weight matrices). The paper notes that this "appears to be more efficient than KAN" in terms of raw parameter count, but "KANs usually require much smaller $N$ than MLPs, which not only saves parameters, but also achieves better generalization." For example, the knot theory KAN (shape [17, 1, 14]) has ~200 parameters and outperforms a 300,000-parameter MLP (Table 3).

The 1D special case. For 1D problems ( $n_0 = 1$ ), a KAN with $N=L=1$ reduces to a single spline—the network is literally just a B-spline fit to the data. This is a useful sanity check: KANs generalize splines to multivariate functions via the layered composition of 1D edge functions.

Grid Extension: Progressive Accuracy Improvement

Grid extension is the mechanism that allows KANs to improve their approximation accuracy without retraining from scratch by refining the B-spline grid. This is fundamentally different from how MLPs improve accuracy (training a larger model from scratch) and is more analogous to mesh refinement in finite element methods.

The procedure (Section 2.4): Suppose we have a trained KAN with coarse grid $G_1$ intervals and corresponding B-spline coefficients $c_i$ ( $i = 0, \ldots, G_1+k-1$ ). We want to extend to a finer grid with $G_2 > G_1$ intervals with coefficients $c'_j$ ( $j = 0, \ldots, G_2+k-1$ ). The new coefficients $c'_j$ are initialized by solving a least-squares problem (Equation 2.16):

$\{c'_j\} = \arg\min_{\{c'_j\}} \mathbb{E}_{x \sim p(x)} \left[ \left( \sum_{j=0}^{G_2+k-1} c'_j B'_j(x) - \sum_{i=0}^{G_1+k-1} c_i B_i(x) \right)^2 \right]$

where $p(x)$ is the distribution of input activations to that edge function (estimated from the training data), $B_i(x)$ are the old coarse-grid B-spline basis functions, $B'_j(x)$ are the new fine-grid basis functions, and the expectation is over $x \sim p(x)$ . The minimization finds the fine-grid coefficients $c'_j$ that make the new spline approximate the old spline as closely as possible in the mean-squared sense over the data distribution.

What this computes operationally: Given the old coarse spline (defined by coefficients $c_i$ on grid $G_1$ ) and the new fine grid (with $G_2$ intervals and corresponding basis functions $B'_j$ ), we solve for the coefficients $c'_j$ that minimize the expected squared difference between the fine spline and the coarse spline, evaluated at points drawn from the empirical distribution of inputs to that edge. This is a linear least-squares problem that can be solved efficiently. After solving, the fine-grid spline is functionally almost identical to the old coarse-grid spline (it's the best $L_2$ approximation of the coarse spline in the span of the fine basis functions), but now it has more degrees of freedom to be further optimized.

Why this form: The least-squares initialization ensures continuity of the approximation across grid extensions. Without it, simply setting $c'_j = 0$ for the new fine-grid coefficients would introduce a discontinuity in the function approximation, potentially increasing the loss and undoing previous training progress. The least-squares initialization allows the fine-grid KAN to start from the same functional approximation as the coarse-grid KAN, and then gradient descent can further reduce the loss by adjusting the fine-grid coefficients. This is analogous to $h$ -refinement in finite elements, where a coarse solution is interpolated onto a finer mesh to serve as an initial guess for the finer solve.

Training dynamics with grid extension (Figure 2.3). The paper demonstrates training dynamics for a toy $f(x,y) = \exp(\sin(\pi x) + y^2)$ with a $[2, 5, 1]$ KAN. Training starts with $G=3$ and the grid is extended to higher values ( $5, 10, 20, 50, 100, 200, 500, 1000$ ) every 200 LBFGS steps. The loss curves (Figure 2.3 top-left) show staircase-like behavior: the training loss drops sharply after each grid extension, then plateaus until the next extension. The test loss (Figure 2.3 bottom-left) shows a U-shape: it decreases with increasing $G$ up to an optimal point (around $G \approx 50$ —which the paper notes roughly matches the interpolation threshold where parameters equal data points: $15G \approx 1000$ training points), then increases due to overfitting.

Scaling law from grid extension (Figure 2.3 bottom-left). For the $[2,1,1]$ KAN (which is the minimal architecture that can exactly represent the toy function), test RMSE scales approximately as $G^{-3}$ for the root-mean-squared error (and roughly $G^{-4}$ for the root-median-squared error). The theoretical prediction from Theorem 2.1 is $G^{-4}$ , and the suboptimal $G^{-3}$ scaling for the mean is attributed to "boundary effects" and a few outlier samples with large errors.

External vs. internal degrees of freedom. The paper introduces this conceptual distinction specifically to articulate what grid extension enables. The connectivity of the computation graph (the shape $[n_0, n_1, \ldots, n_L]$ ) determines the external degrees of freedom—these govern the compositional structure (which variables get combined, whether the function has a generalized additive structure, etc.). The grid resolution $G$ determines the internal degrees of freedom—these govern how accurately each univariate component function is approximated. MLPs only have external dofs (no concept of grid), while splines only have internal dofs (no compositional structure). KANs have both, which is why they can learn both the compositional structure and the accurate 1D functions.

Theorem 2.1: Approximation Theory and Scaling Laws

Theorem 2.1 provides the theoretical guarantee for KANs' approximation power. The statement (Equation 2.15) is:

$\|f - (\Phi^G_{L-1} \circ \Phi^G_{L-2} \circ \cdots \circ \Phi^G_1 \circ \Phi^G_0)(x)\|_{C^m} \leq C G^{-k-1+m}$

for any $0 \leq m \leq k$ , where $f$ is the target function, $\Phi^G_l$ are the KAN layers with grid size $G$ and B-spline order $k$ , $\|\cdot\|_{C^m}$ is the $C^m$ -norm (measuring the maximum magnitude of derivatives up to order $m$ ), and $C$ is a constant depending on $f$ and its representation.

What the theorem assumes: The target function $f$ admits a KAN representation $f = (\Phi_{L-1} \circ \cdots \circ \Phi_0)(x)$ where each component function $\Phi_{l,i,j}$ is $(k+1)$ -times continuously differentiable—i.e., the function has a smooth Kolmogorov-Arnold representation. This is the key assumption that prevents the pathological non-smooth inner functions that historically doomed the theorem.

What the theorem claims: Under this smoothness assumption, there exists a KAN with finite grid size $G$ (approximating each edge function with $k$ th-order B-splines) such that the approximation error in the $C^m$ norm decays as $G^{-k-1+m}$ . For the $L^\infty$ norm ( $m=0$ ), the error decays as $G^{-k-1}$ . For cubic splines ( $k=3$ ), this predicts a $G^{-4}$ scaling of the error, which translates to a scaling exponent $\alpha = k+1 = 4$ for the test RMSE $\ell$ as a function of parameters $N$ (since $N \propto G$ ): $\ell \propto N^{-4}$ .

Why this is significant: The bound is independent of input dimension $n$ . The constant $C$ may depend on $n$ (the paper acknowledges this and "leaves the discussion of the dependence of the constant on the dimension as a future work"), but the asymptotic rate $G^{-k-1}$ does not. This means KANs can, in principle, beat the curse of dimensionality for functions that admit smooth KA representations, because the error decays as a function of the 1D grid resolution, not the ambient dimension. This contrasts with standard approximation theory for MLPs (Sharma & Kaplan), which gives $\alpha = (k+1)/d$ (curse of dimensionality), and with Michaud et al.'s bound $\alpha = (k+1)/d^*$ where $d^*=2$ is the maximum arity of binary operations in the computational graph.

The proof sketch: The proof relies on classical 1D B-spline approximation theory: a $(k+1)$ -times differentiable function can be approximated by $k$ th-order B-splines on a grid of size $G$ with error $O(G^{-k-1})$ in the $C^m$ norm. By applying this to each edge function individually and composing the bounds through the layers (accounting for the fact that composition with bounded smooth functions preserves the approximation order), the total error accumulates as the sum of per-layer errors, each of order $G^{-k-1+m}$ , giving the overall bound $C G^{-k-1+m}$ (where $C$ absorbs the number of layers and the smoothness constants).

Neural scaling law interpretation. The paper connects this to the neural scaling laws literature by claiming $\alpha = 4$ as the theoretical scaling exponent for KANs with cubic splines. Compared to other theoretical predictions— $\alpha = 1$ for ReLU MLPs on general functions ( $k=1$ , $d$ arbitrary), $\alpha = 2$ from Michaud et al. (maximum arity $d^*=2$ with $k=1$ ), $\alpha = m/2$ from Poggio et al.—KANs achieve "the largest and best scaling exponent compared to other works" (Section 2.3). The experiments in Figure 3.1 empirically confirm that KANs approach this $\alpha = 4$ bound on the five synthetic examples with known smooth KA representations, while MLPs plateau well before reaching even the slower $\alpha = 4/d$ bound.

Relation to the Kolmogorov-Arnold theorem (KAT) vs. Universal Approximation Theorem (UAT). The paper distinguishes KAT from UAT: UAT guarantees existence of an approximating MLP but provides no bound on how the required width scales with error $\epsilon$ (and, the paper notes, "N has been shown to grow exponentially with $d$ in some cases"). KAT, under the smoothness assumption, provides an explicit rate $G^{-k-1}$ independent of dimension. The tradeoff is that KAT assumes the function admits a smooth KA representation—a stronger assumption than mere continuity, but one the paper argues is reasonable for "functions we encounter in practice and in science."

Simplification Pipeline: Making KANs Interpretable

The simplification pipeline transforms a trained, overparameterized KAN into a compact, interpretable form that reveals the function's compositional structure and potentially its symbolic formula. This is operationalized through four sequential techniques.

Sparsification (L1 + Entropy Regularization)

Unlike MLPs, which have scalar weights that can be L1-regularized to induce sparsity, KANs have functions on edges. So the paper defines the L1 norm of an activation function $\phi$ as its average absolute magnitude over a batch of $N_p$ inputs (Equation 2.17):

$|\phi|_1 \equiv \frac{1}{N_p} \sum_{s=1}^{N_p} |\phi(x^{(s)})|$

where $x^{(s)}$ are the input values flowing through that edge during training. This measures how "active" the function is—if $\phi$ is nearly zero everywhere, its L1 norm is small.

For a full KAN layer $\Phi$ with $n_{\text{in}}$ inputs and $n_{\text{out}}$ outputs, the L1 norm is the sum over all edge functions (Equation 2.18):

$|\Phi|_1 \equiv \sum_{i=1}^{n_{\text{in}}} \sum_{j=1}^{n_{\text{out}}} |\phi_{i,j}|_1$

What this computes: The per-edge L1 norm $|\phi|_1$ is the expected absolute value of $\phi(x)$ under the empirical distribution of its inputs. It's large if $\phi$ produces large-magnitude outputs (indicating the edge is important for the computation) and small if $\phi$ is near zero (indicating the edge is inactive). The layer norm $|\Phi|_1$ sums these over all edges in the layer.

Entropy regularization (Equation 2.19). The paper finds L1 alone insufficient for sparsification and adds an entropy term:

$S(\Phi) \equiv -\sum_{i=1}^{n_{\text{in}}} \sum_{j=1}^{n_{\text{out}}} \frac{|\phi_{i,j}|_1}{|\Phi|_1} \log\left(\frac{|\phi_{i,j}|_1}{|\Phi|_1}\right)$

What this computes: The entropy of the distribution of normalized edge magnitudes $\frac{|\phi_{i,j}|_1}{|\Phi|_1}$ across all edges in the layer. If all edges have equal magnitude, the entropy is high ( $\log(n_{\text{in}} n_{\text{out}})$ ). If one edge dominates and others are near zero, the entropy is low. Minimizing this entropy (by including it in the loss with a positive coefficient) encourages the network to concentrate magnitude in a few important edges while driving others to zero—this is sparsification.

Why both L1 and entropy: L1 regularization alone tends to shrink all edge functions uniformly, producing many small-but-nonzero edges rather than truly sparse ones. The entropy term penalizes "distributed" magnitude and favors concentration, making the network genuinely sparse rather than just small-everywhere. In the ablation (Appendix C, Figure C.1), the paper shows that without entropy penalty, "there are many duplicate functions" and sparsification fails to produce a clean interpretable graph.

Total training objective (Equation 2.20):

$\ell_{\text{total}} = \ell_{\text{pred}} + \lambda \left( \mu_1 \sum_{l=0}^{L-1} |\Phi_l|_1 + \mu_2 \sum_{l=0}^{L-1} S(\Phi_l) \right)$

where $\ell_{\text{pred}}$ is the prediction loss (MSE for regression, cross-entropy for classification), $\lambda$ controls the overall regularization strength, and $\mu_1 = \mu_2 = 1$ are relative weights (set equal by default, making L1 and entropy equally weighted). The sum runs over all $L$ KAN layers.

Visualization

After training with sparsification, the KAN is visualized as a computation graph where each edge is rendered as a plot of its activation function $\phi_{l,j,i}(x)$ over its input range. The transparency of each edge is set proportional to $\tanh(\beta A_{l,i,j})$ where $A_{l,i,j} = |\phi_{l,j,i}|_1$ is the edge's magnitude and $\beta = 3$ is a scaling factor. This means edges with small magnitude appear transparent (faded out) while important edges are fully opaque, allowing a human to "stare at a KAN" and immediately see which connections matter. The paper contrasts this with MLP interpretability, which requires post-hoc feature attribution methods: "instead of using feature attribution methods (which are great methods), one can instead simply stare at visualizations of KANs" (Section 4.3).

Pruning

After sparsification, some nodes may remain that have some incoming or outgoing edges but are not genuinely necessary. Pruning removes nodes at the whole-node level (not individual edges). For each node $(l, i)$ (the $i$ th neuron in layer $l$ ), two scores are computed (Equation 2.21):

$I_{l,i} = \max_k (|\phi_{l-1,i,k}|_1), \quad O_{l,i} = \max_j (|\phi_{l+1,j,i}|_1)$

where $I_{l,i}$ is the maximum incoming edge magnitude (from any node in layer $l-1$ to node $i$ ), and $O_{l,i}$ is the maximum outgoing edge magnitude (from node $i$ to any node in layer $l+1$ ). A node is considered important if both $I_{l,i}$ and $O_{l,i}$ exceed a threshold $\theta = 10^{-2}$ (by default). Nodes that fail this criterion—meaning they have either negligible input or negligible output—are pruned (removed from the network along with all their incident edges). The rationale: a node must be both receiving non-negligible signal and producing non-negligible signal to be functionally relevant. Pruning is done automatically; the network's shape shrinks as a result.

Symbolification

Once the network is pruned to a minimal form, the learned activation functions can be examined. In many cases, they resemble known symbolic functions (linear, quadratic, sine, exponential, logarithmic, etc.). The symbolification step replaces a learned spline edge function $\phi(x)$ with an affine-transformed symbolic function: $y \approx c \cdot f(ax + b) + d$ , where $f$ is the symbolic function (e.g., $\sin$ , $\exp$ , $x^2$ ) and $a, b, c, d$ are affine parameters (scale and shift of input and output).

The fitting procedure: Given a trained edge function, a user can call fix_symbolic(l, i, j, f) to specify that edge $(l, i, j)$ should be represented as the symbolic function $f$ . The system then: (1) collects samples of inputs $x$ and outputs $y$ from that edge by running training data through the network, (2) fits the affine parameters $a, b, c, d$ such that $y \approx c \cdot f(ax + b) + d$ via "iterative grid search of $a, b$ and linear regression" (for $c$ and $d$ ), and (3) replaces the spline with this parametrized symbolic form. After symbolification, the network can be further fine-tuned to optimize only the affine parameters.

Automatic symbolification. The function suggest_symbolic(l, i, j) compares the learned activation function against a library of candidate symbolic forms (e.g., linear, quadratic, cubic, exponential, logarithmic, sine, cosine, tanh, sigmoid, square root, absolute value) and returns the best matches. auto_symbolic() applies the top-1 suggestion to all edges automatically. However, the paper emphasizes that the manual mode—where the user inspects the visualizations and decides which functions to fix—is often more powerful because it allows the human to inject domain knowledge (e.g., recognizing a cosh in the Anderson localization example).

Symbolic formula extraction. After all edges are symbolified, the function symbolic_formula() uses SymPy to compute the exact symbolic expression of the output node by composing all the symbolic edge functions through the summation nodes. The user obtains a formula like 1.0*exp(1.0*y^2 + 1.0*sin(3.14*x)) for the toy example (Section 2.5.2), which can then be simplified further (rounding coefficients, canceling terms) to recover the ground-truth formula.

The human-in-the-loop interaction model (Section 2.5.2 and Figure 2.4). The paper proposes a specific workflow for scientific discovery with KANs, illustrated through the hypothetical user Alice:

Train with sparsification on an overparameterized KAN (e.g., $[2, 5, 1]$ for a 2-input function). The sparsification regularization causes most edges to shrink.
Prune automatically, removing inactive nodes. In the toy example, this reduces $[2, 5, 1]$ to $[2, 1, 1]$ , and the three remaining edge functions are now visualized.
Inspect and symbolify: The user stares at the three remaining activation curves and recognizes them as $\sin$ , $x^2$ , and $\exp$ . They call fix_symbolic on each edge.
Further train only the affine parameters, achieving machine-precision loss.
Output the symbolic formula via SymPy.

The paper explicitly contrasts this with symbolic regression: SR is all-or-nothing ("either return a success or a failure in the end without outputting interpretable intermediate results"), while KANs provide a continuous, inspectable, debuggable process where "users have more control over KANs as compared to SR due to KANs' transparency."

4. Key Insights and Innovations

Innovation 1: Reframing the Kolmogorov-Arnold Theorem from Pathological Curiosity to Practical Architecture Through Depth Generalization

The most fundamental intellectual move in this paper is not the architectural design itself—learnable activations on edges have been explored before—but the reframing of the Kolmogorov-Arnold representation theorem from a mathematical dead end into a viable foundation for deep learning. This required overturning a consensus that had held for over three decades.

What the field believed before this paper: The Kolmogorov-Arnold theorem had been "sentenced to death in machine learning" (Section 2.1) following Girosi and Poggio's influential 1989 argument that the theorem is "irrelevant" for neural networks because its inner functions can be non-smooth and even fractal. This created a deeply entrenched belief: the theorem is theoretically elegant but practically useless because the univariate functions it requires are unlearnable. Prior attempts to use the theorem for neural networks (Sprecher and Draghici, 2002; Köppen, 2002; Lin and Unbehauen, 1993; Lai and Shen, 2021; Leni et al., 2013; Fakhoury et al., 2022) all "stuck with the original depth-2 width-(2n+1) representation" (Section 1)—the exact form from the 1950s theorem—and many predated backpropagation, producing empirical results that were "rather limited or artificial toy experiments" (Section 5).

The conceptual move that changes everything: The paper's pivotal insight is that the theorem's pathological reputation comes from its shallow form, and that depth can substitute for smoothness. A function that requires non-smooth or fractal 1D components in a 2-layer [n, 2n+1, 1] representation might admit perfectly smooth, spline-approximable 1D components in a 3-layer or deeper representation. The example in Equation (2.13)—a 4-variable function that "may not admit a 2-Layer KAN with smooth activations" but can be "smoothly represented by a [4, 2, 1, 1] KAN which is 3-Layer"—makes this concrete. This is not just "make it deeper and it works better"; it is a fundamental reconceptualization of what the Kolmogorov-Arnold representation is: not a fixed formula but a representational principle whose smoothness properties improve with depth, analogous to how deeper circuits can compute the same Boolean function with smoother gates.

Why this reframing is fundamental rather than incremental: The paper is not proposing a small modification to the theorem. It is proposing that "to the best of our knowledge, there is not yet a 'generalized' version of the theorem that corresponds to deeper KANs" (Section 2.2)—and then acting as if such a generalization exists, building deep networks whose theoretical justification is an empirical hypothesis rather than a proven theorem. This is a bold intellectual stance: the paper declares that the 2-layer version studied for decades is merely the base case of a much richer hierarchy of Kolmogorov-Arnold representations distinguished by depth, and that this hierarchy has fundamentally different smoothness properties than the base case. The evidence in Figure 3.1 (bottom-right) supports this: a 2-layer [4, 9, 1] KAN "behaves much worse than the 3-Layer KAN (shape [4, 2, 2, 1])" on the same function, confirming that deeper KA representations can be both more expressive and smoother.

The diagnostic move from worst-case to typical-case thinking: The paper explicitly invokes a physicist's mindset: "The philosophy here is close to the mindset of physicists, who often care more about typical cases rather than worst cases. After all, our physical world and machine learning tasks must have structures to make physics and machine learning useful or generalizable at all" (Section 2.1). This is not just rhetorical flourish—it is a methodological stance that legitimizes ignoring the fractal worst-case functions that killed the theorem's reputation. By arguing that most functions in science have "smooth and sparse compositional structures" that "potentially facilitat[e] smooth Kolmogorov-Arnold representations," the paper carves out a domain of applicability where the theorem's promise can be realized, even if general continuous functions remain pathological.

Innovation 2: The External/Internal Degrees of Freedom Dichotomy as a Unifying Framework for Understanding Function Approximation Architectures

The paper introduces a conceptual vocabulary—external vs. internal degrees of freedom—that reframes the entire landscape of function approximation methods. This distinction is arguably more valuable than the KAN architecture itself, because it provides a lens through which to understand why different model classes succeed or fail on different types of functions.

The dichotomy: External degrees of freedom are the parameters governing the compositional structure—which variables interact, through what computational graph, in what hierarchical arrangement. Internal degrees of freedom are the parameters governing how accurately each atomic univariate component function is approximated. MLPs have only external dofs (their weight matrices define the compositional graph, but their fixed activations can only approximate 1D functions through compositions of, say, ReLU segments—requiring many parameters for even simple functions like sin or exp). Splines have only internal dofs (they can nail any smooth 1D function with high accuracy, but they have no mechanism for composing multivariate functions—hence the curse of dimensionality). KANs have both.

What this reframes: Before this paper, the MLP-vs-spline comparison was apples-to-oranges. MLPs worked in high dimensions but struggled with even simple univariate functions; splines excelled at 1D approximation but failed in high dimensions. The external/internal dof language reveals that these are not independent drawbacks but symmetric failures: each model class is missing one of two necessary capabilities. This diagnostic move transforms the question from "which architecture is better?" to "how can we build an architecture with both capabilities?"—and KANs emerge as the natural answer.

Why this is more than just a taxonomy: The external/internal distinction has predictive power. It explains why KANs beat the curse of dimensionality on functions with smooth KA representations (Theorem 2.1): the external dofs (the layered composition) handle the high-dimensional structure, while the internal dofs (the B-spline grids) handle the 1D accuracy, and the asymptotic error bound depends only on the internal grid resolution $G$ , not the input dimension $n$ . It explains why grid extension works (Section 2.4): it increases internal dofs while keeping external dofs fixed, allowing progressive accuracy improvement without altering the learned compositional structure. It even explains the U-shaped test loss in Figure 2.3: at the interpolation threshold, the internal dofs saturate the data while the external dofs remain appropriate.

The implications for architecture design beyond KANs: This framework suggests a general design principle for function approximators: any architecture that hopes to handle high-dimensional functions efficiently must separate the learning of compositional structure (requiring something like the layered connectivity of neural networks) from the learning of univariate accuracy (requiring something like splines or other high-resolution 1D approximators). The paper's Learnable Activation Networks (LANs, Appendix B) are an intermediate point on this spectrum—they have external dofs (MLP-like weight matrices) and internal dofs (learnable spline activations on nodes), but the weight matrices "leave too many degree of freedoms" (Appendix B.2) and make interpretability difficult, suggesting that placing learnable activations on edges (eliminating weight matrices entirely) is what enables the clean separation of concerns.

Innovation 3: A Neural Scaling Law Independent of Input Dimension—Theoretical Promise with Empirical Corroboration

The paper makes a striking theoretical claim: under the assumption of a smooth Kolmogorov-Arnold representation, KANs achieve a scaling exponent $\alpha = k+1 = 4$ that is independent of the input dimension, asymptotically faster than any previously established bound for neural networks. This is not just a better constant—it is a qualitative change in how error scales with model size.

The prior scaling law landscape: Before this work, the dominant theoretical frameworks for neural scaling laws all suffered from some form of dimensionality dependence. Sharma & Kaplan's intrinsic dimension theory gives $\alpha = (k+1)/d$ , explicitly cursed by dimension. Michaud et al.'s maximum arity theory gives $\alpha = (k+1)/d^* = (k+1)/2$ for functions expressible with binary operations—better, but still a constant independent of the ambient dimension only under strong compositional assumptions. Poggio et al.'s compositional sparsity theory gives $\alpha = m/2$ for $C^m$ functions, which is dimension-independent but scales only with smoothness order $m$ , not with the approximation order $k$ of the function class. All of these are upper bounds that may not be tight, and empirically, MLPs often plateau well before reaching even these theoretical limits (as shown in Figure 3.1).

What KANs achieve differently: Theorem 2.1 gives $\alpha = k+1 = 4$ for cubic splines, under the assumption that the target function admits a smooth KA representation. The bound is dimension-independent because the error decomposes into 1D spline approximation errors composed through bounded smooth maps—the $C^m$ norm of the composed error is bounded by the sum of per-layer errors, each of which is a 1D spline approximation error scaling as $G^{-k-1+m}$ . The network's total parameter count scales as $N \propto G$ (since dominant parameters are in the B-spline coefficients), giving $\ell \propto N^{-4}$ . The empirical evidence in Figure 3.1 shows KANs approaching this $\alpha = 4$ line on five synthetic examples with known smooth KA representations, while MLPs fail to saturate even the slower $\alpha = 4/d$ bound.

Why this is a fundamental advance rather than an incremental improvement: The implication—if smooth KA representations exist broadly for real-world functions—is that KANs can, in principle, scale to high-dimensional problems with dramatically fewer parameters than MLPs. The paper does not claim this is proven for general functions (the dependence of the constant $C$ on dimension is "left as future work"), but even the possibility of dimension-independent scaling is a conceptual breakthrough. It means the curse of dimensionality is not a fundamental barrier to function approximation—it is an artifact of the MLP architecture's inability to decompose functions into univariate components. The paper reframes the question from "how fast can we approximate arbitrary functions?" (answer: slowly, due to COD) to "how fast can we approximate functions with compositional structure?" (answer: much faster, possibly dimension-independently).

The honest caveats that strengthen the claim: The paper is transparent about limitations: the empirical scaling is $G^{-3}$ (root-mean-squared) rather than the theoretical $G^{-4}$ , attributed to boundary effects and outlier samples (Figure 2.3 bottom-left). The constant $C$ is uncharacterized and may grow with dimension. The assumption of a smooth KA representation is unverified for most real-world functions. But rather than weakening the contribution, this transparency creates a clear research program: characterize when smooth KA representations exist, bound the dimension-dependence of $C$ , and close the gap between theoretical $\alpha = 4$ and empirical $\alpha = 3$ . The paper has opened an asymptotic regime that was previously not even hypothesized as achievable.

Innovation 4: Interpretability as an Architectural Property Rather Than a Post-Hoc Extraction Problem

The paper fundamentally reframes the interpretability problem in deep learning. Rather than treating interpretability as something you extract from a trained black box through post-hoc analysis (saliency maps, feature attribution, sparse autoencoders, circuit analysis), KANs make interpretability an architectural property: the model's learned computation is directly visible by construction.

The dominant paradigm before KANs: The mechanistic interpretability (MI) community, as the paper notes, has been predominantly "passive"—"focusing on understanding existing neural networks trained with standard methods" (Section 5). Even the "active" MI subfield, which designs architectures or training procedures to encourage interpretability, has largely worked within the MLP paradigm: trying to make weight matrices sparse, or activations monosemantic, or circuits disentangled. But the fundamental opacity remains: a weight matrix is a matrix of numbers, and even a sparse one requires interpretation—which combination of inputs produces which feature? The paper provides a stark contrast: in KANs, there are no weight matrices. Every parameter is a point on a 1D curve that you can literally plot and look at.

What makes this a conceptual innovation rather than just a visualization trick: The simplification pipeline (sparsification → pruning → symbolification → formula extraction) is not just a set of tools—it is a new paradigm for human-model interaction in scientific discovery. The hypothetical user Alice (Sections 2.5.2, 4.4) does not run a black-box algorithm and receive a formula. She trains a KAN, stares at the activation curves, forms hypotheses about what functions they represent, tests those hypotheses by fixing symbolic forms, retrains, and iteratively converges to a symbolic expression. The model is a collaborator in a conversation conducted in the language of functions. The paper's framing—"KAN as a 'language model' for AI + Science" (Section 6)—captures this: just as LLMs communicate in natural language, KANs communicate in the native language of science (functions), enabling a mode of interaction where the human and the model jointly construct understanding.

Why this matters beyond symbolic regression accuracy: The paper explicitly contrasts KANs with symbolic regression (SR) methods (Section 2.5.2, Section 4.4). SR is brittle—it "either return[s] a success or a failure in the end without outputting interpretable intermediate results"—and opaque—"it is impossible for Alice to 'debug' or interact with the underlying process of symbolic regression." KANs provide a continuum of interpretability: even before symbolification, the raw spline curves convey information about monotonicity, oscillation frequency, asymptotic behavior, and which input variables matter. Even if the final symbolic formula is never extracted, the scientist has learned something about the functional relationship. And when the function contains a component not in any symbol library (e.g., a Bessel function $J_0$ in Figure 4.1(d)), SR fails entirely while KANs "can use splines to approximate it numerically anyway" and the scientist can recognize the characteristic wiggly shape.

The evidence that this paradigm works: The paper demonstrates the human-KAN collaboration paradigm on three real scientific problems of increasing complexity: knot theory signature prediction (Section 4.3, supervised mode), unsupervised discovery of knot invariant relations (Section 4.3, unsupervised mode), and Anderson localization mobility edge extraction (Section 4.4). In the MAAM case, the paper explicitly documents the iterative human-in-the-loop process across multiple steps (Figure 4.6, Table 5), where the human's choices of which functions to symbolify lead to progressively simpler formulas (from 97.1% accuracy auto-symbolic to 95.4% accuracy with quadratic assumptions on $p$ ), trading off accuracy for simplicity at a granularity that SR cannot match.

The deeper implication for AI + Science: The paper argues that KANs enable a shift from "AI scientists" (fully automated discovery) to "AI that helps scientists" (collaborative discovery). The rationale is profound: "The intrinsic difficulty of (fully automated) AI scientists is that it is hard to make human preferences quantitative" (Section 6). Different scientists in different fields have different notions of what makes a formula "simple" or "interpretable"—a condensed matter physicist might consider $\cosh$ natural while a high-energy physicist prefers polynomial forms. KANs accommodate this heterogeneity by letting the scientist impose their own simplicity criteria through the symbolification choices, rather than baking a one-size-fits-all simplicity metric into an automated search. This is not just a practical convenience—it is a philosophical stance on what scientific discovery requires: human judgment about what constitutes a meaningful explanation, which cannot be fully automated.

Innovation 5: The Diagnostic Identification of Verifier Over-Optimization as the Central Bottleneck—Wait, No. Let Me Recalibrate.

This paper doesn't deal with verifiers. The actual fifth innovation—if we consider it sufficiently distinct from the architectural innovations above—is the demonstration that locality of basis functions can architecturally solve catastrophic forgetting, but this is more of a promising preliminary result than a fully developed contribution. The stronger framing is:

Innovation 5: Locality as an Architectural Solution to Catastrophic Forgetting, Not an Algorithmic Workaround

The paper demonstrates that KANs exhibit natural immunity to catastrophic forgetting in a continual learning setting (Section 3.5, Figure 3.4), and identifies the mechanism as the locality of B-spline basis functions. This is conceptually distinct from the dominant approaches to continual learning, which are algorithmic (EWC, replay buffers, progressive networks) rather than architectural.

The dominant paradigm: Most continual learning methods treat catastrophic forgetting as an optimization problem to be solved through constraints on weight updates (elastic weight consolidation), data replay, or architectural expansion. These are add-ons to architectures that are fundamentally non-local—MLPs with global activation functions like ReLU, where "any local change may propagate uncontrollably to regions far away, destroying the information being stored there" (Section 3.5).

What KANs do differently: Because each B-spline basis function $B_i(x)$ is non-zero only on a local interval $[t_{-k+i}, t_{i+1}]$ , updating the coefficient $c_i$ in response to new data affects the function $\phi(x)$ only in that local region. Data points in far-away regions of the input space leave $c_i$ untouched (their gradient contribution is zero because $B_i(x) = 0$ there). This means that when new data arrives in a previously unseen region of the input space (e.g., phase 2 training on a new Gaussian peak in Figure 3.4), the model can adapt to it without disturbing what it learned about previously seen regions. The paper explicitly frames this as mimicking biological brains: "A key difference between artificial neural networks and human brains is that human brains have functionally distinct modules placed locally in space. When a new task is learned, structure re-organization only occurs in local regions responsible for relevant skills, leaving other regions intact" (Section 3.5).

Where this sits on the incremental-to-fundamental spectrum: This is the paper's most preliminary contribution—tested only on "an extremely simple example" (1D regression with 5 Gaussian peaks), with the authors acknowledging that "it remains unclear whether our method can generalize to more realistic setups, especially in high-dimensional cases where it is unclear how to define 'locality.'" It is better understood as a proof of concept for a research direction rather than a solved problem. The conceptual significance is that it identifies an architectural property (local basis functions) that can naturally provide a property (catastrophic forgetting resistance) that the field has been engineering through algorithmic means. If this generalizes to higher dimensions (perhaps through careful design of the input-space partitioning), it would represent a fundamental shift in how we approach continual learning—from patching non-local architectures to using inherently local ones.

The connection to the paper's broader thesis: This innovation reinforces the paper's central argument that the choice of architectural primitive matters in ways that go beyond accuracy benchmarks. The same B-spline parametrization that enables internal degrees of freedom, grid extension, and interpretability also provides locality that mitigates catastrophic forgetting. It is not a separate contribution but a demonstration of the coherence of the KAN design: multiple desirable properties flow from the same architectural choice (learnable splines on edges, no global weight matrices), suggesting that this choice captures something fundamental about function approximation that the MLP design misses.

5. Experimental Analysis

Evaluation Methodology

Dataset. The experiments span four categories of datasets: (1) five synthetic toy examples with known smooth Kolmogorov-Arnold representations, including a univariate Bessel function, a 2D exponential-sine, the product $xy$ , a 100-dimensional generalized additive model, and a 4D nested compositional function (Section 3.1); (2) 15 special functions from scipy.special common in math and physics (Section 3.2, Table 1); (3) the Feynman dataset (Feynman_no_units), containing physics equations from Feynman's textbooks, filtered to problems with at least 2 variables (Section 3.3, Table 2); and (4) a Poisson PDE with zero Dirichlet boundary conditions on $\Omega = [-1, 1]^2$ , with ground truth solution $u = \sin(\pi x) \sin(\pi y^2)$ (Section 3.4). For the knot theory application (Section 4.3), the dataset consists of 18 knot invariants (17 inputs + signature as the target), sourced from prior work by Davies et al. For Anderson localization (Section 4.4), data is generated from quasiperiodic tight-binding models by randomly sampling model parameters, computing energy eigenvalues, and calculating the fractal dimension of eigenstates.

Base model. All KAN experiments use the proposed architecture with cubic B-splines (order $k=3$ ), SiLU basis functions, and trainable scale parameters $w_b, w_s$ . MLP baselines use ReLU, Tanh, or SiLU activations. Both KANs and MLPs are trained with the LBFGS optimizer for 1,800 total steps. The paper does not use pretrained models—all networks are trained from scratch on each task. KAN widths are typically set to 5 for sweeps, with depths in $\{2, 3, 4, 5, 6\}$ ; MLPs similarly sweep widths of 5 or 100 and depths in $\{2, 3, 4, 5, 6\}$ . For the knot theory task, a [17, 1, 14] KAN (G=3, k=3) with approximately 200 parameters is compared against a 4-layer width-300 MLP from Davies et al. with approximately $3 \times 10^5$ parameters (Table 3).

Metrics. The primary metric across all regression tasks is test RMSE (root mean squared error), reported as a function of model parameter count to construct Pareto frontiers. For classification tasks (knot theory signature prediction, Anderson localization phase classification), test accuracy is used. For PDE solving (Section 3.4), both L2 error squared and H1 error squared (energy norm, which includes derivative errors) are reported. For image fitting (Appendix B.3), PSNR (peak signal-to-noise ratio) is measured. In the continual learning experiment (Section 3.5), predictions are visualized qualitatively rather than quantified with a specific metric.

Baselines. The paper compares KANs against several families of MLPs:

MLPs with fixed activation functions (ReLU, Tanh, SiLU), varying depths (2–6) and widths (5 or 100), trained with LBFGS for 1,800 steps. These serve as the primary accuracy and scaling baseline throughout Sections 3.1–3.4.
SIREN (implicit neural representations with periodic sine activations, Sitzmann et al., 2020) for the image fitting task in Appendix B.3.
Deepmind's MLP (4-layer, width-300, ~3×10⁵ parameters) for the knot theory signature prediction task (Table 3), sourced from Davies et al., 2021.
Human-constructed KANs for the Feynman dataset (Table 2), where the authors manually design KAN shapes expected to represent known formulas. These are compared against auto-discovered (pruned) KANs.
KANs without pruning (fixed shapes with width 5, depths 2–6) as an ablation to quantify the benefit of sparsification and pruning in Sections 3.2 and 3.3.

Generation budget / compute accounting. Compute is measured by number of model parameters for the Pareto frontier analyses in Figures 3.1, 3.2, D.1, and for the efficiency comparisons in Table 3 and Figure 3.3. For grid extension experiments (Figure 2.3), the relevant budget is the number of grid intervals $G$ , which controls the internal degrees of freedom. Training is done for a fixed number of LBFGS steps (1,800 total, with grid extensions every 200 steps) across all methods. Training time comparisons are reported in Figure 2.3 (bottom-right), showing that KAN training time scales favorably with grid size $G$ , though the paper notes KANs are "usually 10x slower than MLPs, given the same number of parameters" (Section 6). For the continual learning experiment, compute is not explicitly budgeted—the point is qualitative behavior rather than efficiency.

Cross-validation / statistical protocol. For KAN experiments in Sections 3.2 and 3.3, each hyperparameter combination (shape, regularization strength $\lambda$ , grid schedule) is run with 3 random seeds, and the best model (minimal KAN shape achieving test RMSE < 10⁻², or lowest test loss) is reported. For the PDE experiment (Section 3.4), hyperparameters $n_i = 10000$ , $n_b = 800$ , and $\alpha = 0.01$ are fixed across all models. For the knot theory unsupervised learning experiment (Section 4.3), 200 networks are trained with $\lambda \in \{10^{-2}, 10^{-3}\}$ and $\text{seed} \in \{0, 1, \ldots, 99\}$ , and results are clustered into three groups (Figure 4.4). No explicit train/validation/test splits are described for most experiments—the paper typically reports test-set performance on held-out data points, but dataset sizes and split ratios are not specified with the exception of the knot theory task (where test accuracy is reported on a held-out set) and the toy grid extension example (1,000 training samples).

Main Quantitative Results

Synthetic Functions with Known KA Representations (Section 3.1, Figure 3.1)

The headline result is that KANs approach the theoretical scaling exponent $\alpha = 4$ (test RMSE $\propto N^{-4}$ where $N$ is parameter count) on all five synthetic examples, while MLPs plateau at substantially higher loss levels and scale much more slowly (roughly following or failing to reach the slower $\alpha = 4/d$ bound predicted by Sharma & Kaplan).

On the univariate Bessel function $f(x) = J_0(20x)$ (Figure 3.1, top-left): A depth-2 KAN achieves test RMSE below $10^{-5}$ with approximately $10^3$ parameters and continues improving toward $10^{-7}$ at $10^5$ parameters, closely tracking the $\alpha = 4$ theoretical line. MLPs of depths 2–5 with ReLU activations plateau at test RMSE between $10^{-4}$ and $10^{-3}$ , with depth providing minimal improvement. The best MLP (depth 5) at $10^5$ parameters achieves roughly $2 \times 10^{-4}$ test RMSE, approximately two orders of magnitude worse than the KAN at the same parameter count.

On the 2D function $f(x, y) = \exp(\sin(\pi x) + y^2)$ (Figure 3.1, top-center): A depth-2 KAN rapidly descends below $10^{-4}$ test RMSE at roughly $3 \times 10^3$ parameters and approaches $10^{-6}$ at $10^5$ parameters, tracking the $\alpha = 4$ line. MLPs of all depths plateau between $10^{-3}$ and $10^{-2}$ , with no MLP configuration breaking below $10^{-3}$ even at $10^5$ parameters. The KAN achieves approximately 100× lower test RMSE than the best MLP at matched parameter counts.

On the product $f(x, y) = xy$ (Figure 3.1, top-right): The KAN follows $\alpha = 4$ scaling, reaching $10^{-7}$ test RMSE at $10^5$ parameters. MLPs show a curious behavior: depth-2 MLPs initially track the slower $\alpha = 4/d = 2$ line but plateau around $10^{-4}$ ; deeper MLPs (depth 4–5) do worse than shallower ones, with the depth-5 MLP plateauing near $10^{-2}$ . This is a notable reversal—standard intuition says deeper networks are more expressive, but here the architectural mismatch (MLPs cannot efficiently represent multiplication) causes deeper MLPs to overfit or optimize poorly.

On the high-dimensional generalized additive model $f(x_1, \ldots, x_{100}) = \exp(\frac{1}{100} \sum_{i=1}^{100} \sin^2(\pi x_i / 2))$ (Figure 3.1, bottom-left): This is the most dramatic comparison. A depth-2 KAN with shape [100, 1, 1] precisely matches the function's structure (100 independent sine-squared terms, summed, exponentiated) and tracks the $\alpha = 4$ line, reaching $10^{-4}$ test RMSE at approximately $10^4$ parameters. MLPs of all depths completely fail: all plateau above $10^{-1}$ test RMSE regardless of parameter count. The gap is approximately three orders of magnitude at matched parameter counts. This demonstrates that when the function's compositional structure aligns with the KA representation (generalized additive form), KANs effectively reduce the problem to 1D function approximation and avoid the curse of dimensionality entirely, while MLPs—lacking any mechanism to decompose the function into independent per-dimension components—cannot learn the structure at all.

On the 4D compositional function $f(x_1, x_2, x_3, x_4) = \exp(\frac{1}{2}(\sin(\pi(x_1^2 + x_2^2)) + \sin(\pi(x_3^2 + x_4^2))))$ (Figure 3.1, bottom-right): The critical comparison here is between a 2-layer KAN [4, 9, 1] and a 3-layer KAN [4, 2, 2, 1]. The 3-layer KAN tracks the $\alpha = 4$ line and reaches $10^{-7}$ test RMSE at $10^4$ parameters. The 2-layer KAN [4, 9, 1] performs dramatically worse, reaching only about $10^{-2}$ test RMSE—approximately 5 orders of magnitude worse—and following the slower $\alpha \approx 1$ line. This provides direct evidence for the paper's central claim that depth matters for smoothness of KA representations: the same function admits a smooth 3-layer representation but requires a non-smooth (and therefore hard-to-approximate-with-splines) 2-layer representation. MLPs of all depths plateau between $10^{-2}$ and $10^{-1}$ , with none matching even the 2-layer KAN.

Special Functions (Section 3.2, Figure 3.2, Table 1)

Across all 15 special functions, KANs achieve better Pareto frontiers than MLPs in the (number of parameters, RMSE) plane. The paper reports both the minimal KAN shape that achieves test RMSE < 10⁻² and the best KAN shape (lowest test loss).

For the Bessel function of the first kind $jv(x, y)$ : The minimal KAN shape [2, 2, 1] achieves test RMSE $4.93 \times 10^{-3}$ ; the best KAN [2, 3, 1, 1, 1] achieves $1.64 \times 10^{-3}$ ; the MLP baseline achieves $5.52 \times 10^{-3}$ . The KAN achieves approximately 3.4× lower test RMSE than the MLP.

For the incomplete elliptic integral of the second kind ellipeinc(x, y): The minimal [2, 2, 1, 1] KAN achieves $8.36 \times 10^{-5}$ test RMSE; the best KAN (same shape) matches this; the MLP achieves $3.04 \times 10^{-4}$ . The KAN achieves approximately 3.6× lower test RMSE.

For spherical harmonics $sph\_harm(0, 1, x, y)$ : A [2, 1, 1] KAN achieves $2.21 \times 10^{-7}$ test RMSE; the MLP achieves $1.25 \times 10^{-6}$ . The KAN is approximately 5.7× more accurate.

For the associated Legendre function (m=0) $lpmv(0, x, y)$ : The [2, 2, 1] KAN achieves $5.25 \times 10^{-5}$ test RMSE vs. $1.74 \times 10^{-2}$ for the MLP—approximately 330× lower error.

However, on some functions, the advantage is smaller or KANs struggle: for the modified Bessel function of the first kind $iv(x, y)$ , the best KAN [2, 4, 3, 2, 1, 1] achieves $9.28 \times 10^{-3}$ test RMSE, while the MLP achieves $1.07 \times 10^{-2}$ —only a marginal improvement (1.15×). The KAN shape is also unusually deep and wide, suggesting this function does not admit a compact KA representation.

Figure 3.2 shows the consistent pattern: KAN Pareto frontiers lie below and to the left of MLP frontiers across all 15 functions, meaning KANs achieve lower error at the same parameter count, or alternatively, require fewer parameters to achieve the same error. The pruned KANs (auto-discovered shapes) are reported in Table 1, with many achieving surprisingly compact shapes—e.g., sph_harm(0, 1) and sph_harm(0, 2) both reduce to [2, 1, 1] KANs, suggesting these spherical harmonics admit a simple additive-multiplicative decomposition in a KA representation.

Feynman Dataset (Section 3.3, Table 2, Figure D.1)

The Feynman dataset results show more nuanced behavior. The paper's own summary is that "MLPs and KANs behave comparably on average" (Section 3.3), and the Pareto frontier plot (Figure D.1 in Appendix D) confirms this: KAN and MLP frontiers overlap substantially for many of the 28 equations, unlike the clear separation seen for special functions.

However, several specific results stand out:

For $I.6.2$ : $f(\theta, \sigma) = \exp(-\theta^2 / 2\sigma^2) / \sqrt{2\pi\sigma^2}$ : The pruned KAN [2, 2, 1] achieves test RMSE $2.86 \times 10^{-5}$ , while the MLP achieves $1.45 \times 10^{-4}$ —approximately 5× lower error.

For $I.9.18$ (gravitational potential): The human-constructed KAN [6, 4, 2, 1, 1] achieves $1.48 \times 10^{-3}$ ; the pruned KAN [6, 4, 1, 1] achieves $8.62 \times 10^{-3}$ (worse); the MLP achieves $1.59 \times 10^{-3}$ . Here, the human-constructed KAN slightly outperforms the MLP, but the pruned KAN is worse—showing that pruning can sometimes remove useful structure when the target function requires a specific compositional architecture.

For $I.16.6$ (relativistic velocity addition $f(u, v) = (u+v)/(1+uv)$ ): The human-constructed KAN shape is [2, 2, 2, 2, 2, 1]—5 layers deep—reflecting the expected decomposition through squared terms and division. However, the auto-discovered pruned KAN achieves shape [2, 2, 1]—only 2 layers—with test RMSE $1.19 \times 10^{-3}$ , which is comparable to the MLP's $6.20 \times 10^{-4}$ . This is the "rapidity trick" the paper highlights: $u+v)/(1+uv) = \tanh(\text{arctanh } u + \text{arctanh } v)$ can be realized by a 2-layer KAN using tanh and arctanh as the learned activation functions (Section 3.3). The KAN independently discovers a more compact representation than the human construction.

For $I.30.5$ : $\arcsin(a/n)$ : A [2, 1, 1] KAN achieves $3.49 \times 10^{-5}$ test RMSE, vs. the MLP's $9.45 \times 10^{-5}$ —approximately 2.7× better, and the KAN shape is trivially minimal.

Across the Feynman dataset, the pruned KAN shapes are "usually smaller than our human constructions" (Section 3.3), with many reducing to 2-layer or 3-layer architectures. This suggests that KA representations can be more efficient than even domain-expert humans anticipate.

PDE Solving (Section 3.4, Figure 3.3)

The Poisson equation experiment demonstrates substantial advantages for KANs in physics-informed neural networks:

A 2-layer width-10 KAN [2, 10, 1] achieves L2 error squared of approximately 10⁻⁷ at convergence, while a 4-layer width-100 MLP [2, 100, 100, 100, 1] achieves only approximately 10⁻⁵—a 100× improvement in accuracy. Even more strikingly, the KAN achieves this with only about 10² parameters vs. the MLP's 10⁴ parameters, making it simultaneously more accurate and more parameter-efficient by a factor of approximately 100×.

The training dynamics (Figure 3.3, left panels) show that the KAN converges faster: it reaches 10⁻⁴ L2 error squared within approximately 50 LBFGS steps, while the width-100 MLP takes approximately 150 steps to reach the same error. The width-10 MLP (matched width to the KAN) plateaus around 10⁻²—two orders of magnitude worse than the KAN.

The scaling laws (Figure 3.3, right panels) show that KAN L2 error follows the $\alpha = 4$ scaling line against parameter count, while MLP error scales much more slowly and plateaus. The H1 error (which includes derivative errors, important for PDE solutions) shows an even wider gap: the KAN achieves approximately 10⁻² H1 error squared at 10² parameters, while the best MLP (depth-5) requires 10⁵ parameters to reach comparable H1 error.

However, the paper includes an important caveat: "The ground truth being a symbolic formula might be an unfair comparison for MLPs since KANs are good at representing symbolic formulas" (Section 3.4). This is a genuine limitation—the PDE solution is $\sin(\pi x) \sin(\pi y^2)$ , which is exactly the kind of function KANs excel at representing.

Continual Learning (Section 3.5, Figure 3.4)

The toy continual learning experiment shows qualitative behavior rather than quantitative metrics. A 1D regression task with 5 Gaussian peaks is presented sequentially (data around each peak is introduced one phase at a time). After each phase:

KAN predictions (middle row): The model accurately fits the current peak while preserving the fit on previously learned peaks. By phase 5, all 5 peaks are correctly modeled. The spline adapts locally—only the B-spline coefficients whose basis functions overlap with the new data region are updated.
MLP predictions (bottom row): After each new phase, the MLP remodels the entire function, destroying previously learned peaks. By phase 2, the first peak is already forgotten; by phase 5, only the most recent peak (phase 5) is correctly modeled, with all previous peaks lost.

This is a proof-of-concept rather than a benchmark result—no quantitative forgetting metrics are reported, the problem is 1D, and the paper acknowledges that "it remains unclear whether our method can generalize to more realistic setups, especially in high-dimensional cases where it is unclear how to define 'locality'" (Section 3.5).

Knot Theory: Supervised Signature Prediction (Section 4.3, Table 3, Figure 4.3)

An extremely small [17, 1, 14] KAN (G=3, k=3) with approximately 200 parameters achieves 81.6% test accuracy on predicting knot signatures, compared to 78.0% for Deepmind's 4-layer width-300 MLP with approximately 3 × 10⁵ parameters (Table 3). This is simultaneously a 1.036× accuracy improvement and a 1,500× parameter reduction.

The interpretability result is equally striking: by visualizing the trained KAN (Figure 4.3, left), it is immediately apparent that signature depends primarily on the real part of meridional translation $\mu_r$ , with secondary dependence on imaginary meridional translation $\mu_i$ and longitudinal translation $\lambda$ . This is discovered without post-hoc feature attribution methods—the transparency of the edge activations directly encodes variable importance.

Ablation experiments (Figure 4.3, right) confirm this: using only $\mu_r$ as input achieves 65.0% test accuracy; $\mu_i$ alone achieves only 43.8%; all three variables ( $\mu_r, \mu_i, \lambda$ ) achieve 78.2% accuracy with a [3, 1, 14] KAN.

The paper notes a subtle difference from Davies et al.: "they find that signature is mostly dependent on $\mu_i$ , while we find that signature is mostly dependent on $\mu_r$ " (Section 4.3). This discrepancy is attributed to "subtle algorithmic choices" and is not resolved—it remains an open question which dependence is correct or whether both capture different aspects of the relationship.

Knot Theory: Symbolic Formula Discovery (Section 4.3, Table 4)

KANs with different shapes produce a Pareto frontier of symbolic formulas trading off simplicity and accuracy:

Formula B ([3, 1] KAN): Complex expression involving sin, absolute value, and exponential terms; test accuracy 62.6%, $r^2 = 0.837$ with signature.
Formula C ([3, 1, 1] KAN): Slightly more complex; test accuracy 71.9%, $r^2 = 0.871$ .
Formula D ([3, 2, 1] KAN): The most complex and accurate; test accuracy 84.0%, $r^2 = 0.947$ with signature and $r^2 = 0.997$ with the Deepmind-discovered formula A ( $\lambda \mu_r / (\mu_r^2 + \mu_i^2)$ ).
Formula E ([3, 2, 1] KAN + Padé approximation): By imposing a multivariate Padé structure, KAN discovers $\frac{4.76 \lambda \mu_r}{3.09 \mu_i + 6.05 \mu_r^2 + 3.54 \mu_i^2}$ , which is structurally similar to Deepmind's formula A; test accuracy 82.8%, $r^2 = 0.946$ .
Formula F ([3, 1] KAN/[3, 1] KAN, manual numerator/denominator): Involves only $\mu_r$ and $\lambda$ (ignoring $\mu_i$ ); test accuracy 77.8%, $r^2 = 0.925$ .

The key finding is that KANs can rediscover the known formula $\lambda \mu_r / (\mu_r^2 + \mu_i^2)$ (with $r^2 = 0.946$ ) when appropriate inductive biases (Padé form) are imposed, and can independently discover simpler formulas (Formula F) that use only two variables with only modest accuracy loss.

Knot Theory: Unsupervised Relation Discovery (Section 4.3, Figure 4.4)

In unsupervised mode, 200 trained KANs ([18, 1, 1] with sparsification) cluster into three groups, each discovering a known mathematical relation:

Signature dependence group: Identifies that signature, $\mu_r$ (real meridional translation), and $\lambda$ (longitudinal translation) are dependent variables—rediscovering the supervised result.
Cusp volume relation: Discovers $-\log V + \log \mu_r + \log \lambda = 0$ , which simplifies to $V = \mu_r \lambda$ . The paper notes this is "true by definition"—a sanity check that validates the unsupervised approach.
Short geodesic and injectivity radius: Finds that $g_r$ (real part of short geodesic) and injectivity radius have a linear correlation, consistent with the known mathematical result that $2r$ (where $r$ is the injectivity radius) upper-bounds $g_r$ .

The paper's honest assessment: "The good news is that the results discovered by KANs are probably reliable; the bad news is that we have not discovered anything new yet" (Section 4.3). This is a refreshingly candid admission that the method rediscovers known results but has not yet produced novel mathematics.

Anderson Localization: Mobility Edge Discovery (Section 4.4, Figure 4.5, 4.6, Table 5)

For the Mosaic Model (MM): A [2, 1] KAN with the second activation manually set to sigmoid (classification task) and the $\lambda$ activation fixed to linear (reflecting the assumption $g(\lambda, E) = \lambda - h(E)$ ) successfully extracts the mobility edge $\lambda = h(E)$ as a 1D curve. The learned curve (Figure 4.5, middle) closely matches the ground truth phase diagram (Figure 4.5, top). The qualitative visualization (Figure 4.5, bottom) shows the learned $h(E)$ function.

For the Generalized Aubry-André Model (GAAM, Figure 4.6a): A [4, 2, 1, 1] KAN trained with sparsification achieves 98.7% accuracy. The visualization (step 1) shows that $\phi$ (phase factor) is completely ignored by the network, correctly identifying that the mobility edge is phase-independent. After automatic symbolic snapping to linear/quadratic functions (step 2), the network achieves 98.9% accuracy and produces the symbolic formula (Table 5, GAAM-KAN auto):

$1.52E^2 + 21.06\alpha E + 0.66E + 3.55\alpha^2 + 0.91\alpha + 45.13\lambda - 54.45 = 0$

This is close to the ground truth $\alpha E + 2\lambda - 2 = 0$ (99.2% accuracy). The paper notes that by "cross[ing] out some small terms and snap[ping] coefficient to small integers," Alice can recover the true formula.

For the Modified Aubry-André Model (MAAM, Figure 4.6b-c, Table 5): This is the most complex case and demonstrates the iterative human-KAN collaboration:

Step 1: A [4, 2, 1, 1] KAN achieves only 75% accuracy (unacceptable), so Alice upgrades to [4, 3, 1, 1] and achieves 98.4%.
Step 2: After pruning to [4, 2, 1, 1], the KAN maintains 97.7% accuracy. The visualization shows that $\phi$ is again ignored.
KAN auto-symbolic: Produces a complex formula (Table 5, MAAM-KAN auto) with 97.1% accuracy, involving nested sin, arctan, and exponential terms.
Step 3 (manual snapping): Alice manually sets all non- $p$ activations to linear/quadratic; retrains to 97.7% accuracy with a simpler formula.
Step 4A (cosh assumption): Alice guesses the dependence on $p$ is cosh and fixes it; achieves 96.6% accuracy with formula close to the ground truth $\lambda \cosh(p) = E + 1$ .
Step 4B (quadratic assumption): Alice pursues simplicity and assumes $p$ dependence is quadratic; achieves 95.4% accuracy with a simpler but less accurate formula.

This sequence demonstrates the paper's core collaboration thesis: KANs provide a continuum of interpretability where the human can trade off accuracy and simplicity at a granular level, choosing which functional forms to impose based on domain knowledge or simplicity preferences.

Ablation Studies and Robustness Checks

Pruned vs. unpruned KANs on Feynman dataset (Table 2, columns 3–6): On I.6.2, the pruned KAN [2, 2, 1] achieves test RMSE $2.86 \times 10^{-5}$ vs. the unpruned KAN's $4.60 \times 10^{-5}$ —pruning not only reduces parameters but improves accuracy, suggesting the sparsification penalty acts as beneficial regularization. However, on I.9.18, the pruned KAN [6, 4, 1, 1] achieves $8.62 \times 10^{-3}$ (worse than the human-constructed KAN's $1.48 \times 10^{-3}$ and the unpruned KAN's $6.56 \times 10^{-3}$ ), indicating pruning can sometimes remove necessary structure.

Effect of entropy regularization on sparsity (Appendix C, Figure C.1): Without entropy penalty ( $\mu_2 = 0$ ), the network retains many duplicate activation functions and sparsification fails to produce a clean interpretable graph. With entropy, the network concentrates magnitude in a few edges (ideally 3 for the $f(x, y) = \exp(\sin(\pi x) + y^2)$ toy example), making the pruned graph maximally interpretable. This justifies the paper's claim that "L1 [alone is] insufficient for sparsification of KANs."

Effect of overall regularization strength $\lambda$ (Appendix C, Figure C.1): Too small $\lambda$ leads to insufficient sparsification (many active edges remain); too large $\lambda$ over-prunes and removes necessary edges. There is an optimal intermediate $\lambda$ that achieves the desired 3-edge network.

Effect of random seed (Appendix C, Figure C.1): Some seeds produce pruned networks larger than the minimal [2, 1, 1] target, indicating that sparsification + pruning can converge to local optima in architecture space. The paper does not report statistics over seeds (e.g., what fraction of seeds achieve the minimal shape), leaving uncertainty about the reliability of the pruning procedure.

Effect of grid size G on interpretability (Appendix C, Figure C.1): When $G$ is too small (e.g., $G=3$ ), each activation function lacks expressivity, so the network tends to "use the ensembling strategy" (multiple edges collaborate to approximate one function), making interpretation harder. Larger $G$ (e.g., $G=20$ ) gives each edge sufficient capacity to independently represent a meaningful univariate function, aiding interpretability.

Effect of spline order k (Appendix C, Figure C.1): The piecewise polynomial order has "only a subtle effect on interpretability" and "behaves a bit like the random seeds which do not display any visible pattern." The paper uses $k=3$ (cubic) throughout all experiments.

Grid extension vs. training larger models from scratch (Figure 2.3): Grid extension produces staircase-like loss curves where each extension yields a sharp loss drop. Training a [2, 1, 1] KAN (the true minimal shape) produces better final test loss than a [2, 5, 1] KAN (overparameterized), with the interpolation threshold delayed to larger $G$ due to fewer parameters. This demonstrates that knowing the true KAN shape matters: overparameterization leads to earlier overfitting.

KAN depth comparison on 4D example (Figure 3.1, bottom-right): A 2-layer KAN [4, 9, 1] achieves approximately 10⁻² test RMSE, while a 3-layer KAN [4, 2, 2, 1] achieves approximately 10⁻⁷—a 100,000× improvement. This is the strongest evidence for the paper's claim that deeper KA representations can be smoother and therefore more learnable.

Human-constructed vs. auto-discovered KAN shapes (Table 2): In most Feynman equations, auto-discovered (pruned) KAN shapes are smaller than human-constructed ones. For I.6.2, the human construction [2, 2, 1, 1] is pruned to [2, 2, 1]; for I.16.6, [2, 2, 2, 2, 2, 1] prunes to [2, 2, 1]; for II.11.27, [2, 2, 1, 2, 1] prunes to [2, 1, 1]. This suggests humans overestimate the depth needed for these formulas because they think in terms of explicit algebraic decompositions (e.g., multiplication requires $xy = ((x+y)^2 - (x-y)^2)/4$ ), while KANs can discover more exotic univariate functions (like rapidity transformations) that compress the representation.

MLP activation function comparison (Figure D.1): Across the Feynman dataset, MLPs with ReLU, Tanh, and SiLU activations show overlapping Pareto frontiers—the choice of fixed activation function makes little systematic difference. This reinforces the claim that fixed activations, regardless of which one, are the fundamental limitation.

KAN pruning for special functions (Table 1, Figures F.1, F.2): Many special functions admit surprisingly compact KA representations. For example, sph_harm(0, 1) and sph_harm(0, 2) both reduce to [2, 1, 1]—a single hidden layer with one neuron. The ellipj function requires [2, 3, 2, 1, 1, 1] for best accuracy but can achieve test RMSE < 10⁻² with only [2, 2, 1]. This suggests that approximate KA representations can be substantially more compact than exact ones.

LAN vs. KAN interpretability (Appendix B.2, Figure B.2): Learnable Activation Networks (MLPs with learnable spline activations on nodes, but with weight matrices retained) are "much less interpretable" than KANs on the synthetic examples where KANs perfectly recover the compositional structure. The paper conjectures that "weight matrices leave too many degree of freedoms, making learnable activation functions too unconstrained" and that "getting rid of linear weight matrices (by having learnable activations on edges, like KANs) is necessary for interpretability."

LAN for image fitting (Appendix B.3, Figure B.3): A SIREN network (MLP with sine activations) can be adapted to LAN by adding trainable spline components to the sine activations. The LAN achieves higher PSNR than the baseline SIREN, and fine-tuning a pretrained SIREN as a LAN (initializing spline components to zero) provides further improvement, with only a ~4% parameter increase (G=5, N=128).

Critical Assessment

Do the experiments support the claim that KANs have faster neural scaling laws than MLPs?

The evidence in Figure 3.1 strongly supports this claim for the five synthetic examples with known smooth KA representations. KANs empirically approach the theoretical $\alpha = 4$ scaling line, while MLPs plateau at substantially higher error. However, the claim's generality is limited in several important ways:

The functions were chosen because they have known smooth KA representations. The paper constructs examples (1)–(5) precisely because they can be represented by KANs with shapes [1,1], [2,1,1], [2,2,1], [100,1,1], and [4,2,2,1]—all of which the authors know in advance. This is a best-case scenario for KANs. The scaling advantage on these examples demonstrates that if a function admits a smooth KA representation, KANs scale better than MLPs. But the paper does not test functions without smooth KA representations, leaving open the crucial question: what is the empirical scaling behavior on functions where the KA representation is unknown or non-smooth? The special function and Feynman experiments partially address this, but those are lower-dimensional and produce more mixed results.

The theoretical $\alpha = 4$ is not actually achieved for the mean RMSE. Figure 2.3 (bottom-left) shows test RMSE scaling as $G^{-3}$ for the mean squared error, not $G^{-4}$ as predicted by Theorem 2.1. The paper attributes this to "boundary effects" and outliers, noting that the median squared error scales closer to $G^{-4}$ . This gap between theoretical and empirical scaling—even on the simplest toy example—suggests that the $\alpha = 4$ bound is an asymptotic ideal not realized in practice with finite grids and LBFGS optimization. The synthetic examples in Figure 3.1 show KAN lines tracking the $\alpha = 4$ theoretical line, but on log-log plots with only ~3 orders of magnitude in parameter count, distinguishing $\alpha = 3$ from $\alpha = 4$ requires careful measurement that the paper does not provide (no fitted slopes, no confidence intervals).

The MLP baseline training may not be optimal for those architectures. Both KANs and MLPs are trained with LBFGS for a fixed 1,800 steps. If MLPs require more steps, different learning rates, or different optimization strategies to achieve their best scaling, the comparison undercounts MLP capability. The paper acknowledges this: "Note that we have adopted the vanilla setup where both KANs and MLPs are trained with LBFGS without advanced techniques, e.g., switching between Adam and LBFGS, or boosting. We leave the comparison of KANs and MLPs in advanced setups for future work" (Section 3.1). This is a genuine limitation—the claim of superior scaling might not hold if MLPs are given their best optimization treatment.

Does the evidence support the claim that KANs outperform MLPs in accuracy with fewer parameters?

This claim has strong support but with important caveats about when the advantage manifests:

On synthetic functions with compositional structure (Figure 3.1): Yes, dramatically. The 100D additive model shows a ~10³× RMSE gap at matched parameters. The 4D compositional example shows that a 3-layer KAN achieves ~10⁻⁷ RMSE at 10⁴ parameters where the best MLP achieves ~10⁻² at 10⁵ parameters.

On special functions (Figure 3.2, Table 1): Yes, consistently. Every one of the 15 special functions shows KAN Pareto frontiers dominating MLP frontiers. The improvements range from marginal (iv, ~1.15×) to dramatic (sph_harm, ~5.7×; lpmv(m=0), ~330×). However, note that these are all 2-variable functions. The advantage may shrink in higher dimensions—the paper doesn't test, say, 10-variable special functions.

On Feynman equations (Table 2, Figure D.1): The advantage is inconsistent. The paper's own summary—"MLPs and KANs behave comparably on average"—is accurate. Some equations show KAN advantages (I.6.2, I.30.5); others show MLP advantages or near-parity (I.9.18, I.16.6, II.6.15a). The Feynman equations tend to be simpler (smooth, monotonic dependencies) where MLPs are not severely challenged, so the KAN advantage may only become pronounced when the target function contains features MLPs struggle with (oscillations, sharp transitions, high-order polynomial dependencies).

On the knot theory task (Table 3): The ~200-parameter KAN achieves 81.6% vs. 78.0% for the ~3×10⁵-parameter MLP. This is a ~1,500× parameter reduction for ~1.04× accuracy improvement—clearly supporting the claim. However, a single task with a single train/test split does not establish robust superiority. Moreover, the paper notes (Table 3 footnote) that after the preprint's release, "Prof. Shi Lab from Georgia tech discovered that an MLP with only 60 parameters is sufficient to achieve 80% accuracy." This substantially weakens the claim: if a 60-parameter MLP can achieve comparable performance, the 1,500× parameter advantage may be an artifact of the baseline MLP being massively overparameterized rather than evidence of KANs' inherent efficiency.

On the PDE task (Figure 3.3): The 100× accuracy improvement and 100× parameter reduction are impressive, but the paper's own caveat applies: the ground truth solution $\sin(\pi x) \sin(\pi y^2)$ is exactly the kind of function KANs represent well. A fairer test would be a PDE whose solution does not have an obvious KA decomposition, or a nonlinear PDE (Navier-Stokes, as the paper suggests for future work) where the solution structure is not known to be KA-friendly.

Does the evidence support the claim that KANs are interpretable and useful for scientific discovery?

The evidence for interpretability is the paper's strongest contribution, but its implications for novel scientific discovery remain unproven:

The interpretability pipeline works on synthetic examples (Figure 4.1). The six examples ( $xy$ , $x/y$ , numerical-to-categorical, Bessel function, phase transition, Euclidean distance) all produce visualizations where the compositional structure and univariate functions are clearly readable. This validates the core mechanism.

On known scientific results, KANs rediscover what is already known. The knot theory experiments rediscover: (a) that signature depends primarily on $\mu_r$ and $\lambda$ (known from Davies et al.), (b) the cusp volume relation $V = \mu_r \lambda$ (which is "true by definition"), and (c) the linear correlation between $g_r$ and injectivity radius (a "well-known relation"). The Anderson localization experiments rediscover known mobility edge formulas for MM, GAAM, and MAAM. The paper's candid admission—"The good news is that the results discovered by KANs are probably reliable; the bad news is that we have not discovered anything new yet"—is commendably honest but highlights a gap: the paper demonstrates that KANs can recover known structure, not that they can discover unknown structure. The distinction matters because recovering known results risks confirmation bias—the user knows what to look for and can steer the symbolification process accordingly. The true test would be applying KANs to a dataset where the underlying relationship is genuinely unknown and having domain experts subsequently verify a KAN-discovered formula.

The human-in-the-loop interaction is demonstrated but not systematically evaluated. The MAAM example (Figure 4.6, Table 5) shows Alice making a sequence of choices (cosh vs. quadratic for $p$ dependence), but the paper does not report how many iterations were required, how sensitive the outcome is to user expertise, or whether different users would converge to different formulas. The collaboration paradigm is illustrated, not validated.

The comparison to symbolic regression is qualitative, not quantitative. The paper argues that KANs are superior to SR because they provide a continuum of interpretability and allow user interaction, but no head-to-head comparison is provided (e.g., accuracy of KAN-discovered formulas vs. PySR-discovered formulas on the same Feynman equations). The claim that "SR is brittle and hard to debug" is asserted rather than demonstrated experimentally.

Does the evidence support the claim that KANs avoid catastrophic forgetting?

This claim is supported only by a single 1D toy example (Figure 3.4). The demonstration is visually compelling but lacks quantitative metrics, statistical significance, or generalization to higher dimensions. The paper acknowledges this as "preliminary results on an extremely simple example" and notes that "it remains unclear whether our method can generalize to more realistic setups." The claim in the executive summary—"KANs can naturally work in continual learning without catastrophic forgetting"—overstates what the experiments actually demonstrate. A more accurate characterization would be: "KANs show promising locality-based resistance to catastrophic forgetting in a 1D proof-of-concept, but generalization to practical continual learning settings is untested."

What experiments would have strengthened the paper but were not run?

Large-scale benchmarks. All experiments are on small-scale problems (toy functions, special functions with 2 variables, the 500-question Feynman dataset, a 2D Poisson equation). There are no experiments on standard deep learning benchmarks (CIFAR, ImageNet, language modeling) where MLPs are used as components of larger architectures. The paper's vision of "kansformers" (transformers with KANs replacing MLPs) is entirely speculative without experiments.
Head-to-head comparison with symbolic regression methods. Given that the paper claims KANs are superior to SR for scientific discovery, a quantitative comparison on the Feynman dataset (which SR methods have been benchmarked on) would substantially strengthen the claim.
Systematic difficulty characterization. The paper introduces difficulty bins for functions (synthetic with known KA representations, special functions, Feynman equations) but does not systematically characterize what properties of a function predict KAN advantage. Is it smoothness? Compositional sparsity? Oscillation frequency? Such an analysis would make the claims more predictive and less post-hoc.
Statistical characterization of pruning reliability. What fraction of random seeds produce the correct minimal KAN shape? How sensitive is the discovered architecture to hyperparameters ( $\lambda$ , $\mu_1$ , $\mu_2$ , $G$ )? The paper shows a few representative seeds in Appendix C but does not provide aggregate statistics.
Ablation of the basis function choice. The SiLU basis $b(x) = x/(1+e^{-x})$ is used throughout, but is it essential? How does performance change with identity, zero, or other basis functions? This ablation would clarify whether the basis function is providing meaningful structure or is incidental.
Training time / FLOPs-matched comparisons. The paper acknowledges KANs are 10× slower than MLPs per parameter, but all comparisons are based on parameter count, not training time or FLOPs. A FLOPs-matched comparison (similar to the compute-matched analysis common in scaling law papers) would substantially change the efficiency narrative: if a 1,000-parameter KAN takes 10× longer to train than a 10,000-parameter MLP, the total compute might be comparable.
Confidence intervals on scaling exponents. The scaling law claims (Figure 3.1, Figure 2.3) report point estimates without error bars. Given that these are log-log fits over ~3 orders of magnitude with 3 random seeds, the uncertainty in the fitted slopes could be substantial.
Tests on non-symbolic, non-smooth ground truth functions. The PDE experiment uses a symbolic ground truth. All toy examples have clean formulas. What happens when the ground truth is a solution to a PDE with no closed form, or a function represented only by data with no known compositional structure? This is the regime where scientific discovery actually happens.

Summary of evidential strength

The paper's strongest empirical claims—that KANs can represent compositional functions with high accuracy, that grid extension enables progressive refinement, and that the learned activation functions are visually interpretable—are well-supported by the presented experiments. The scaling law advantage ( $\alpha = 4$ vs. MLP plateau) is demonstrated convincingly on the specific synthetic examples but its generality remains unproven. The scientific discovery claims are better characterized as "KANs can recover known structure from data" rather than "KANs can discover novel scientific laws"—the latter remains aspirational. The efficiency claims (fewer parameters for better accuracy) are well-supported but would benefit from FLOPs-matched comparisons given the 10× training speed penalty. The continual learning claim is preliminary and overclaimed relative to the experimental evidence.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted For and Potentially Prohibitive

The assumption or constraint. The compute-optimal allocation framework relies entirely on estimating prompt difficulty before deciding how to spend the inference budget. The paper's difficulty estimation procedure requires generating 2048 samples per question and either checking ground-truth correctness (oracle bins) or averaging the PRM's final-answer scores (predicted bins), as described in Section 3.2. The authors acknowledge this explicitly: "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2).

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the 2048-sample overhead into the total budget. For a question that would receive, say, 64 generations under the compute-optimal policy, the true cost is 2048 + 64 = 2112 generations—more than 30× the allocated budget. This means the headline efficiency claims are an upper bound that is not achievable in deployment without a substantially cheaper difficulty estimator. In a realistic system, the difficulty estimation cost could dominate the total compute, potentially making the compute-optimal policy more expensive overall than simply applying a uniform best-of-N strategy with a larger budget.

What evidence exists in the paper. No experiment accounts for difficulty estimation cost. The curves in Figures 4 and 8 plot accuracy against the generation budget after difficulty is known, with separate lines for oracle and predicted bins—but neither line includes the 2048-sample estimation overhead. The paper does not measure how accuracy scales with total compute (estimation + execution), nor does it report the fraction of total compute consumed by difficulty estimation at different budget levels. The cross-validation protocol (Section 3.2) splits the 500-question test set per bin, meaning strategy selection is done on ~50 questions per bin per fold—the difficulty estimation cost on these questions is never quantified.

Mitigation status. The paper flags this as "a key avenue for future work" (Section 3.2) and mentions the possibility of "training models to directly predict difficulty of a question" but develops no such model. No experiments with cheaper difficulty proxies (e.g., using only 4–8 samples instead of 2048) are reported. The gap between the current method's practical cost and the reported gains remains unaddressed.

6.2 Hard Problems Remain Essentially Unsolved—Test-Time Compute Cannot Create Capability

The assumption or constraint. The compute-optimal framework assumes the base model has a non-trivial probability of producing correct solutions—i.e., pass@1 is well above zero for the target questions. The paper explicitly identifies difficulty bin 5 (the hardest quintile) as a failure regime: "On the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability" (Section 5.3, discussing Figure 3 right). For the revision model, the same pattern holds: "Bin 5: All ratios produce roughly 2–3% accuracy. No allocation strategy helps" (Section 6, discussing Figure 7 right).

The consequence. Test-time compute amplifies existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help—there are no correct solutions in the proposal distribution to find or refine. This establishes a hard capability ceiling for the approach: problems outside the base model's effective knowledge boundary cannot be solved regardless of inference budget. For these problems, pretraining remains the only viable path. The FLOPs-matched comparison in Section 7 quantifies this: on hard questions (bins 4–5), test-time compute with the smaller model is substantially worse than the ~14× larger model across all $R$ regimes, with deficits ranging from approximately -3.6% to -52.9% depending on $R$ (Figure 9, Figure 1 bar charts).

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% across all methods and budgets. Figure 7 (right) shows bin 5 revision accuracy at 2–3% regardless of the sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line essentially flat near 0–5% across all test-time budgets, while the ~14× larger model's greedy performance (stars) are consistently higher. The gap is starkest for PRM search at $R \gg 1$ : -52.9% relative disadvantage for test-time compute on hard questions (Figure 1, bottom-right bar chart).

Mitigation status. The paper is transparent about this limitation. Section 7 explicitly states that "test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time." The takeaway box in Section 7 provides clear guidance: prefer scaling pretraining when "the problem distribution skews toward genuinely hard problems outside the base model's capability range." However, the paper does not explore whether there are intermediate strategies—e.g., fine-tuning the base model on the specific problem domain before applying test-time compute—that could shift problems from bin 5 to lower bins where test-time compute becomes effective.

6.3 All Results Come from a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the paper—search, revisions, difficulty binning, FLOPs-matched comparisons—uses the MATH benchmark (500 test questions from Hendrycks et al., 2021) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not a demonstrated fact.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration, different error patterns, or different reasoning styles might produce PRM scores that exhibit different over-optimization thresholds. The paper itself notes that the PRM800k dataset (which uses GPT-4 generated solutions) was "largely ineffective" for their PaLM 2 models "likely due to distribution shift" (Section 5.1)—this distribution sensitivity could cut both ways, meaning the specific optimal strategies found for PaLM 2-S* might not transfer to other models.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The 38% correct-to-incorrect reversion rate (Section 6.1) might be higher or lower for other models.
MATH consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains—code generation, logical deduction, scientific question-answering—or to tasks requiring factual knowledge rather than multi-step inference. The paper provides no cross-domain evidence.
The test set of 500 questions, split into five difficulty quintiles of ~100 each, means that per-bin strategy selection (with two-fold cross-validation halving each bin to ~50 questions) is based on very small samples. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed per-bin differences are statistically reliable or could be noise from small sample sizes.

What evidence exists in the paper. No experiments on any dataset other than MATH. No experiments with any model family other than PaLM 2-S* (plus the ~14× larger PaLM 2 variant used only for the FLOPs-matched comparison). The paper does not report standard deviations, confidence intervals, or statistical tests for any of the main results. The cross-validation protocol is described (Section 3.2) but the number of folds (2) and the effective sample size per bin per fold (~50) are not discussed in terms of statistical power.

Mitigation status. The paper does not attempt to address this limitation. No multi-benchmark or multi-model experiments are suggested for future work. The claim of representativeness ("we believe this model is representative") is presented as sufficient justification, which is a significant gap given the generality of the conclusions drawn.

6.4 Sequential Revision Strategies Introduce Serial Latency That Is Not Accounted For in Any Budget

The assumption or constraint. The paper measures test-time compute exclusively in "generations"—the number of complete solutions sampled—which is a reasonable proxy for total FLOPs but ignores wall-clock latency. The compute-optimal policy often allocates budget in a way that requires sequential computation: revision chains where each step conditions on the previous one, or beam search where each expansion round depends on the PRM's evaluation of the previous round.

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes approximately 64× longer wall-clock time than one that runs 128 parallel samples simultaneously (assuming sufficient hardware parallelism for the parallel case). The compute-optimal policy frequently favors sequential allocations, especially on easy problems where sequential revisions dominate (Figure 7, right). For latency-sensitive applications—interactive assistants, real-time decision-making, systems with strict response time SLAs—the sequential-heavy strategies favored by the compute-optimal policy may be impractical regardless of their accuracy advantages. The paper's 4× efficiency gains in terms of generation count could translate to substantially smaller (or even negative) gains in terms of latency-adjusted throughput.

What evidence exists in the paper. The paper does not report wall-clock time for any experiment. The generation budget is the sole unit of compute. There is no discussion of latency constraints, no measurement of per-step latency for sequential vs. parallel strategies, and no comparison of accuracy under latency-matched budgets. Figures 4 and 8, which show compute-optimal curves matching best-of-N at 4× fewer generations, do not indicate what the latency ratio is for the compared strategies.

Mitigation status. This limitation is not acknowledged in the paper. The generation-budget metric is used throughout without caveats about parallelism or latency. The "Takeaway" box in Section 6 (decision tree, Figure 6.1) makes no mention of latency as a consideration. The paper does not suggest latency-aware allocation strategies or pareto-optimal tradeoffs between accuracy and response time.

6.5 The ~14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the FLOPs-Matched Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by ~14× while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge that this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters should be scaled equally: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7).

The consequence. A Chinchilla-optimal ~14× larger model—one that scales both parameters and training tokens to maintain the optimal ratio—would likely outperform a parameter-only-scaled model trained on the same data. This means the pretraining baseline in the FLOPs-matched comparison is weaker than a truly compute-optimal larger model would be. The reported advantages of test-time compute over pretraining—e.g., +27.8% on easy questions at $R \ll 1$ for revisions (Figure 1, top-right bar chart)—might shrink or reverse against a properly compute-optimal larger model. Additionally, the ~14× larger model uses only greedy decoding with no test-time budget of its own. Giving the larger model even a modest test-time compute budget (say, best-of-8 with majority voting) would create a substantially stronger baseline that is never tested.

What evidence exists in the paper. The paper explicitly states the parameter-only scaling choice but provides no comparison to a compute-optimally trained baseline. The $R$ ratio analysis (Section 7) assumes $X = 6 N D_{\text{pretrain}}$ pretraining FLOPs and $Y = 2 N D_{\text{inference}}$ inference FLOPs, which are standard approximations, but the constant factors in these approximations depend on the training recipe. The paper does not discuss how the results might change if the larger model were Chinchilla-trained. The greedy decoding baseline for the larger model is described in Section 7 but never compared against a version with even minimal test-time augmentation.

Mitigation status. The paper acknowledges this as a limitation in the text and defers to future work. No experiments with compute-optimally trained baselines are conducted. The paper does not estimate the magnitude of the potential bias (how much better the Chinchilla-optimal baseline would be). The acknowledgment is honest but does not reduce the uncertainty about whether the reported test-time-compute advantages would survive a fairer comparison.

6.6 The Difficulty Estimation Protocol—Generating 2,048 Samples Per Question—Is the Limitation That Subsumes Several Others

This limitation deserves separate treatment because it cascades into multiple practical concerns beyond the cost issue discussed in Section 6.1, and the paper treats it as the central unsolved deployment problem.

The assumption or constraint. The compute-optimal framework requires a difficulty estimate before strategy selection. The paper's method requires generating 2,048 samples per question and scoring them with the PRM (or checking against ground truth). The authors acknowledge: "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2).

The consequence beyond raw FLOPs. This requirement creates three compounding problems:

First, difficulty estimation cost grows with deployment volume. If you process $Q$ questions and the average test-time budget per question is $N$ generations, the difficulty estimation cost is $2048 \times Q$ generations (amortized), while the useful computation is $N \times Q$ . For any practical deployment where $Q$ is large, the difficulty estimation cost either dominates the total compute (if done per-question) or requires amortization across questions assumed to share a difficulty distribution—but the paper provides no mechanism for batched difficulty estimation.

Second, the PRM-based difficulty estimator itself may suffer from distribution shift. The paper reports that the base-LM PRM underperforms a revision-specific ORM when scoring revision model outputs (Figure 15a), indicating that the PRM's calibration depends on the proposal distribution. If test-time compute strategies change the proposal distribution (e.g., by using the revision model), the difficulty estimate based on the base model's PRM scores may become miscalibrated, creating a circular dependency: you need difficulty to choose the strategy, but the strategy changes the distribution on which difficulty was estimated.

Third, the difficulty bins are coarse and static. The five-quintile discretization assigns the same strategy to all questions within a bin, ignoring within-bin heterogeneity. A question at the easy end of bin 3 and one at the hard end of bin 3 receive identical treatment, even though different strategies might be optimal. The paper does not explore sensitivity to the number of bins, nor does it develop dynamic strategies that adjust mid-computation based on early indicators of difficulty.

What evidence exists in the paper. The paper demonstrates that predicted (non-oracle) difficulty bins perform nearly as well as oracle bins in the search setting (Figure 4) and slightly worse in the revision setting (Figure 8, difference of ~3 percentage points at 256 generations). This validates that PRM-based difficulty estimation is feasible, but does not address the cost question. The paper reports no experiments varying the number of bins, no experiments on dynamic difficulty estimation, and no experiments where difficulty estimation cost is included in the total budget. The cross-domain generalization of the difficulty bins (do questions from a different distribution sort similarly?) is untested.

Mitigation status. The paper explicitly calls for future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). The authors frame the difficulty estimation cost as an "exploration-exploitation tradeoff" and suggest that adaptive methods could subsume difficulty estimation into the solution process. However, no such methods are developed or tested. The difficulty estimation problem is correctly identified as the key deployment bottleneck, but the paper stops at identification without offering even a preliminary solution.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a genuinely new architectural primitive into the deep learning toolbox—not an incremental modification of the MLP, but a clean-slate redesign grounded in a mathematical theorem that the field had prematurely abandoned. The shift is conceptual rather than merely empirical: the paper reframes what a neural network layer is, moving from "linear transformation followed by fixed nonlinearity" to "matrix of learnable univariate functions applied elementwise before summation." This has cascading consequences for how we think about accuracy, interpretability, parameter efficiency, and scaling laws.

It is a paradigm shift in architecture design, but one scoped to function approximation. The paper does not claim KANs will replace transformers or convnets; the scope is explicitly "small-scale AI + Science tasks" and the authors are refreshingly honest about current limitations (10× slower training, no large-scale benchmarks). But within that scope, the shift is fundamental: the paper demonstrates that the MLP's separation of linear weights and fixed activations is not a necessary feature of neural computation but a design choice with specific drawbacks—poor univariate function approximation, lack of fine-graining capability, opacity of learned representations, and vulnerability to catastrophic forgetting. KANs show that an alternative design, fusing the "weight" and "activation" into a single learnable function on each edge, simultaneously addresses all four drawbacks. This is not a collection of independent improvements but symptoms of a common architectural cause that KANs resolve coherently.

The reframing that matters most is the external/internal degrees of freedom dichotomy. Before this paper, MLPs and splines lived in separate worlds: MLPs handled high-dimensional compositional structure but struggled with accurate 1D function approximation; splines excelled at 1D accuracy but choked on dimensionality. The paper's diagnostic—that these are symmetric failures of missing one of two necessary capabilities—creates a conceptual unification. KANs emerge not as a "third option" but as the natural synthesis that completes the space of possible architectures. This reframing makes the design principle explicit: any architecture that hopes to efficiently approximate high-dimensional functions must separate the learning of compositional structure (requiring something like layered connectivity) from the learning of univariate accuracy (requiring something like splines). This principle will influence architecture design beyond KANs.

The paper reconciles the Kolmogorov-Arnold theorem's theoretical promise with its practical reputation. Since Girosi and Poggio's 1989 paper, the KA theorem had been "sentenced to death in machine learning" because its inner functions can be non-smooth and fractal. The paper's resolution is elegant and likely to be enduring: depth substitutes for smoothness. A function requiring pathological 1D components in a shallow 2-layer representation may admit smooth, spline-approximable 1D components in a deeper representation. The empirical evidence in Figure 3.1 (bottom-right)—where a 3-layer KAN achieves ~10⁻⁷ test RMSE while a 2-layer KAN on the same function achieves only ~10⁻²—provides concrete validation. This does not prove that all functions have smooth deep KA representations, but it demonstrates the phenomenon exists and explains why prior shallow attempts failed. The door is now open to a "generalized Kolmogorov-Arnold theorem" that characterizes function classes by their minimum smooth-representation depth—a notion the paper calls "Kolmogorov-Arnold depth" (Section 6).

Interpretability shifts from post-hoc extraction to architectural transparency. The mechanistic interpretability community has largely worked within the MLP paradigm, developing tools to reverse-engineer what black-box networks have learned. KANs bypass this entirely: because the only learnable parameters are 1D functions on edges, the learned computation is directly visualizable by plotting those functions. The paper's demonstration that a [17, 1, 14] KAN with ~200 parameters can reveal variable importance without any feature attribution method (Figure 4.3)—simply by making unimportant edges transparent—is a qualitatively different approach to interpretability. It makes the case that for scientific applications where understanding the function matters as much as predicting it, architectural interpretability is preferable to retroactive explanation. However, this paradigm shift is currently limited to the specific regime where KANs work well (small-scale, compositional functions); whether it generalizes to the messy, high-dimensional functions learned by large transformers is an open question.

The scaling law claim—if it generalizes—would be transformative but remains unproven. Theorem 2.1 provides a dimension-independent scaling exponent $\alpha = 4$ (for cubic splines) under the assumption of a smooth KA representation. The empirical evidence in Figure 3.1 shows KANs approaching this bound on synthetic examples with known KA structure, while MLPs plateau. This is the most provocative claim in the paper because it challenges the conventional wisdom that the curse of dimensionality is an inescapable barrier to function approximation. If smooth KA representations exist broadly for real-world functions—and the paper provides suggestive evidence from special functions and Feynman equations that they might—then KANs could theoretically scale to high-dimensional problems with dramatically fewer parameters than MLPs. But the gap between "synthetic functions we designed to have KA representations" and "arbitrary real-world functions" is vast, and the paper does not bridge it. The practical implication is that this research direction is now open and theoretically motivated, but the burden of proof for generalizability remains.

Follow-Up Research This Work Enables

Cheap difficulty estimation for compute-optimal KAN deployment. The paper's difficulty estimation cost—2,048 samples per input, Section 3.2 of the other paper (not this one—wait, this is a different paper entirely). Let me restart this section properly focused on the KAN paper.

A systematic characterization of when functions admit smooth Kolmogorov-Arnold representations. The paper's central theoretical assumption—that the target function admits a smooth KA representation—is currently verified only for synthetic examples specifically constructed to have one. What is needed is an empirical survey: take a diverse set of functions (special functions, PDE solutions, learned neural networks, real-world datasets), train KANs of varying depths on each, and measure the achieved scaling exponent. A function that admits a smooth KA representation should empirically track the $\alpha = 4$ scaling line; one that does not should plateau like MLPs. This would map out the boundary of KAN applicability. Specifically: train [n, 2n+1, 1] (shallow), [n, various depths] KANs on each function, measure the test RMSE vs. parameter count scaling exponent, and test whether depth reduces the exponent toward $\alpha = 4$ . A negative result—finding that most real-world functions do not admit smooth KA representations even at depth—would substantially narrow KANs' claimed applicability. A positive result would validate the paper's optimism.

Head-to-head comparison of KAN-based symbolic regression against genetic and neural-network SR methods on standard benchmarks. The paper claims KANs are superior to symbolic regression (SR) because they provide a continuum of interpretability and allow human-in-the-loop interaction, but provides no quantitative comparison. A direct experiment: run PySR, Eureka, AI Feynman, and KAN-based SR on the Feynman dataset (which SR methods have been benchmarked on), measuring: (1) recovery rate of the exact ground-truth formula, (2) test RMSE of recovered formulas, and (3) wall-clock time including human interaction (for KANs). The KAN approach would use the full interpretability pipeline: train with sparsification, prune, auto-symbolify or manually symbolify based on visual inspection, retrain affine parameters, extract formula. This would either validate the paper's SR-superiority claim quantitatively or reveal that automated SR matches or exceeds KANs when the formula is in the search space, with KANs offering advantages only when the function contains components outside the SR library (e.g., Bessel functions).

Scaling KANs to large-scale deep learning: replacing MLPs in transformers (the "kansformer"). The paper's vision of "kansformers" is entirely untested. A concrete experiment: take a small transformer (e.g., GPT-2 scale, or a vision transformer like ViT-Tiny), replace the MLP blocks with KAN layers of comparable parameter count, and train on a standard benchmark (CIFAR-10/100 for ViT, WikiText-103 for GPT). Measure: (1) final accuracy/perplexity, (2) training wall-clock time, (3) whether the KAN exhibits the $\alpha = 4$ scaling behavior or plateaus, and (4) whether the learned edge functions in the transformer's KAN blocks reveal interpretable structure in the attention outputs. The key question is whether KANs' advantages (parameter efficiency, accuracy on compositional functions) survive the transition from small-scale function fitting to the messy, high-dimensional, non-symbolic distributions of real-world data. A negative result—KAN transformers failing to match MLP transformers at any parameter count, or training being impractically slow—would significantly narrow the paper's claimed impact. A positive result would open a new architecture family.

Can KANs discover genuinely novel scientific laws, not just rediscover known ones? The paper's most ambitious claim—that KANs enable a new paradigm of AI-scientist collaboration—is supported only by rediscovery of known results (knot invariants, mobility edges). A high-risk, high-reward experiment: apply KANs to a dataset where the underlying relationship is genuinely unknown or only conjectured. Candidates: (1) high-temperature superconductor critical temperature as a function of material properties (the Holy Grail of condensed matter), where domain experts have partial theories but no complete formula; (2) protein folding stability as a function of amino acid sequence features; (3) cosmological parameters as a function of CMB power spectrum features. The protocol would be: train KANs, apply the simplification pipeline, extract candidate symbolic formulas, and have domain experts evaluate whether the formulas: (a) are physically plausible, (b) make testable predictions that differ from existing theories, and (c) survive experimental validation. This is the acid test for KANs as scientific discovery tools. Even a single novel formula—one that domain experts confirm is non-obvious and subsequently validate—would validate the paper's central thesis far more strongly than all the synthetic and rediscovery experiments combined.

Ablation of the specific design choices that make KANs work: which of the two innovations (learnable activations, edge placement) matters more? The paper identifies two changes from MLPs: (i) activation functions are on edges instead of nodes, and (ii) activation functions are learnable instead of fixed. The Learnable Activation Networks (LANs, Appendix B) have (ii) but not (i), and the paper reports they are "less interpretable" and "less accurate than KANs, but still more accurate than MLPs" (Appendix B.1). A systematic ablation would: (1) train LANs, KANs, and MLPs across a range of function classes and dimensions, (2) measure accuracy, parameter efficiency, and interpretability for each, and (3) quantify how much of the KAN advantage comes from each innovation. A finding that most of the advantage comes from learnable activations alone (with edge placement providing only interpretability benefits) would suggest a simpler path to adoption: replace fixed activations with learnable splines in standard architectures without the more radical edge-placement change. This is a low-risk, high-information experiment that would guide architecture design for practitioners.

Rigorous empirical characterization of catastrophic forgetting resistance in higher dimensions. The paper's continual learning demonstration is a single 1D toy example with 5 Gaussian peaks. To determine whether the locality argument generalizes: construct a higher-dimensional continual learning benchmark (e.g., sequential learning of different functions defined on different subregions of $[0,1]^n$ , or sequential learning of different PDE solutions with different boundary conditions). Train KANs and MLPs (with elastic weight consolidation, replay buffers, and other SOTA continual learning methods as baselines) and measure: (1) forgetting on previously learned tasks after each new task, (2) forward transfer (whether learning earlier tasks helps later ones), and (3) whether KANs' locality property degrades with increasing dimension. The paper's hypothesis is that KANs avoid forgetting because B-spline basis functions are local in input space—but in high dimensions, "locality" becomes harder to define because the volume of the input space grows exponentially. If the locality advantage disappears above, say, 5–10 dimensions, the continual learning claim would be revealed as a low-dimensional curiosity rather than a general architectural property. If it persists, KANs could become a preferred architecture for continual learning applications.

Practical Applications and Downstream Use Cases

Physics-informed neural networks (PINNs) for PDE solving with KAN backbones. The paper's Poisson equation experiment (Section 3.4) shows a 2-layer width-10 KAN achieving 100× better accuracy than a 4-layer width-100 MLP (10⁻⁷ vs. 10⁻⁵ MSE) with 100× fewer parameters (10² vs. 10⁴). This has immediate practical implications for any application where PINNs are currently used and accuracy is bottlenecked by the MLP backbone: computational fluid dynamics, solid mechanics, electromagnetic simulation, climate modeling. A KAN-PINN could deliver higher-accuracy solutions with smaller networks, reducing both training time and inference time for PDE surrogate models. The concrete benefit: on problems where ground-truth solutions are known to have compositional structure (which is common in physics—separation of variables, additive potentials, product-form solutions), KANs can exploit that structure while MLPs must learn it from scratch. Practitioners using PINNs today could drop in KANs as a replacement for the MLP layers with minimal workflow changes (the physics-informed loss formulation remains identical).

Interpretable surrogate models for scientific simulation where understanding matters as much as prediction. In many scientific and engineering contexts—drug discovery, materials design, climate projection—researchers need not just accurate predictions but insight into the functional relationship between inputs and outputs. For example, a pharmaceutical chemist wants to know why a particular molecular property depends on certain structural features, not just that it does. KANs' edge-function visualizations and symbolification pipeline enable this: after training on simulation data (e.g., DFT calculations of molecular properties, CFD simulations of aerodynamic drag), the scientist can read off the learned univariate functions and their compositional structure, identify which variables interact, and extract candidate symbolic formulas. The paper's demonstration on Anderson localization mobility edges (Section 4.4)—where KANs recovered known formulas like $\lambda \cosh(p) = E + 1$ and allowed the user to trade accuracy for simplicity—is a template for this workflow. The practical benefit over standard MLP surrogates: when the scientist needs to explain or publish the discovered relationship, a KAN provides a communicable formula rather than a black-box prediction.

Compact model deployment on resource-constrained hardware for function approximation tasks. The knot theory experiment (Table 3) demonstrates a ~200-parameter KAN matching a ~3×10⁵-parameter MLP—a ~1,500× parameter reduction. Even accounting for the fact that a 60-parameter MLP was later found to achieve similar accuracy (Table 3 footnote), the broader pattern across special functions (Figure 3.2) and Feynman equations (Table 2) shows KANs consistently achieving comparable or better accuracy than MLPs with fewer parameters. For deployment scenarios where model size is the binding constraint—embedded systems, edge devices, real-time control with strict memory limits—KANs could replace MLPs in function approximation pipelines (sensor calibration curves, control policy representations, system identification models) with substantial memory savings. The grid extension mechanism (Section 2.4) further enables a deploy-time accuracy-efficiency tradeoff: train a coarse-grid KAN, then refine the grid only for critical regions of the input space without retraining the entire model.

Unsupervised discovery of variable dependencies in scientific datasets. The paper's unsupervised learning mode (Sections 4.2, 4.3) offers a method for identifying groups of dependent variables without specifying input-output splits—which, as the paper notes, has $2^d - 2$ combinatorial possibilities for $d$ variables. The knot theory experiment demonstrated this by rediscovering three known relations among invariants (signature dependence, cusp volume identity, geodesic-injectivity radius correlation) from an [18, 1, 1] KAN trained to distinguish real data from feature-shuffled negatives. This has direct application to exploratory data analysis in scientific domains with many measured variables: genomics (which genes' expression levels are functionally related?), climate science (which atmospheric variables covary?), neuroscience (which neural activity measures are interdependent?). The practical workflow: collect multivariate data, train an unsupervised KAN with sparsification to identify which small subsets of variables the KAN uses to distinguish real from shuffled data, visualize the learned edge functions to hypothesize the form of the dependence, and validate against domain knowledge. The paper's candid finding—"we have not discovered anything new yet"—tempers expectations, but the method provides a principled alternative to correlation-based variable selection that can capture nonlinear dependencies.

When to Prefer This Method

The paper explicitly provides a decision framework for choosing between KANs and MLPs in Figure 6.1 and the accompanying discussion in Section 6. The tradeoff is articulated clearly, so this section is warranted.

Prefer KANs over MLPs when:

Interpretability matters. If the goal is to understand the functional relationship (scientific discovery, model auditing, extracting symbolic formulas), KANs provide directly visualizable learned functions and a symbolification pipeline, while MLPs require post-hoc interpretation tools.
High accuracy on modest-scale problems is the priority. For function fitting, special functions, PDE solving, and similar AI + Science tasks, KANs consistently achieve lower test RMSE than MLPs at matched parameter counts (Figures 3.1, 3.2, 3.3; Tables 1, 2, 3).
The target function is expected to have compositional structure. When there is reason to believe the function decomposes into compositions of univariate operations (generalized additive models, products of independent terms, nested transformations common in physics equations), KANs exploit this structure and achieve faster scaling laws than MLPs.
Training speed is not the binding constraint. KANs are "usually 10x slower than MLPs, given the same number of parameters" (Section 6), and the authors acknowledge they "did not try hard to optimize KANs' efficiency." For offline scientific analysis where training time is measured in hours rather than milliseconds, this penalty is acceptable. For production systems with strict training latency requirements, it may not be.
Continual learning or local adaptation is needed (preliminary). The spline-based locality provides natural resistance to catastrophic forgetting, though this has only been demonstrated on a 1D toy example (Section 3.5, Figure 3.4).

Prefer MLPs over KANs when:

Training speed is critical. The 10× slowdown is the "biggest bottleneck of KANs" (Section 6), and the decision tree in Figure 6.1 advises: "If one wants to train a model fast, one should use MLPs."
The problem is large-scale and lacks known compositional structure. All KAN experiments are on small-scale problems (≤100 input dimensions, ≤1,000 training samples for the grid extension toy example). MLPs are battle-tested on large-scale deep learning; KANs are not.
Hardware or software constraints require standard operations. KANs require B-spline evaluation on every edge, which is not a standard operation in GPU-optimized linear algebra libraries (matmuls are highly optimized; per-element spline evaluations are not). MLPs use operations that are near the peak of hardware efficiency.
The primary metric is wall-clock inference time, not parameter count. KANs may use fewer parameters but the per-parameter computation is more expensive (each edge evaluates a spline rather than multiplying by a scalar). For latency-sensitive inference, the parameter efficiency advantage may not translate to speedup.

The paper explicitly leaves open the possibility of hybrid KAN-MLP architectures (Section 6, point 3), noting that "one can interpolate between activation functions being all the same (MLPs) and all different (KANs)" by grouping edges into "multi-head" clusters that share activation functions. This hybrid space—which the paper does not explore—might capture much of KANs' accuracy advantage while recovering most of MLPs' speed, and is identified as a promising algorithmic direction.