Fisher Simplicity in Kolmogorov-Arnold Networks and Multilayer Perceptrons

2026-09-26

AI Summary

This study addresses the limitation that coefficient sparsity in Kolmogorov–Arnold Networks (KANs) cannot directly serve as a pruning criterion, and investigates the unclear alignment between architectural simplicity and statistical Fisher simplicity. By comparing the Fisher null space properties of dead ReLUs and fixed-basis KANs, this work proposes an effective edge metric based on graph paths, supported by mathematical derivations and controlled experiments integrating Fisher information matrix theory with basis Gram matrix analysis. The findings reveal that Fisher simplicity in multi-layer KANs is governed by data propagation paths rather than coefficient magnitudes. Furthermore, zero effective exposure is shown to accurately identify Fisher null directions, demonstrating that individual coefficient magnitudes are insufficient as a Fisher-based pruning criterion for KANs. These insights refine the theoretical understanding of model simplification.

Abstract

Kolmogorov-Arnold Networks (KANs) are motivated in part by interpretability: their learned edge functions can be inspected, pruned, and reduced to symbolic structure. In a fixed-basis KAN, this makes a small or zero basis coefficient look like a certificate of simplicity, much as a dead rectified linear unit (ReLU) marks unused computation in a multilayer perceptron (MLP). Fisher nullity gives a precise statistical notion: a parameter direction is Fisher-simple exactly when perturbing it is invisible under the task distribution. We study when these architectural and Fisher notions agree. For a dead ReLU unit, they agree: the closed activation region makes the associated score directions vanish. For a fixed-basis KAN, they do not. In the single-layer Gaussian case, the coefficient Fisher matrix is a basis Gram matrix under the input distribution and is independent of the fitted coefficients. In a multilayer KAN, Fisher simplicity is graph-path based: the data must reach a basis atom and its perturbation must propagate through the downstream network. We encode these two conditions in an effective edge measure and, under local dictionary independence and effective-measure nondegeneracy, show that zero effective exposure exactly identifies Fisher-null directions within an edge. Controlled diagnostics confirm that zero coefficients can preserve rank while effective path disconnections remove the predicted directions. Coefficient magnitude alone is therefore not a Fisher-based pruning criterion for KANs.

Problem

• Kolmogorov-Arnold Networks
• Fisher simplicity
• interpretability
• network pruning
• Multilayer Perceptrons

Innovation

• Kolmogorov-Arnold Networks
• Fisher simplicity
• network pruning
• effective edge measure
• interpretability