Combinations of Fast Activation and Trigonometric Functions in Kolmogorov-Arnold Networks

2025-08-15

AI Summary

To address the weak GPU support and low computational efficiency of existing basis functions (e.g., B-splines, RBFs) in Kolmogorov–Arnold Networks (KANs), this work proposes replacing them with GPU-optimized piecewise-linear and trigonometric activation functions—specifically ReLU, sin, cos, and arctan—to construct a novel, high-efficiency KAN architecture. The key contribution is the first integration of non-smooth yet hardware-efficient ReLU with periodic trigonometric functions within the Kolmogorov–Arnold representation framework, preserving theoretical expressive power while substantially improving operator parallelism and memory access efficiency. Experimental results across multiple benchmark tasks demonstrate that the proposed method achieves 1.8–3.2× faster training convergence compared to standard KANs and MLP baselines, with superior or comparable generalization performance.

Technology Category

Machine Learning: Kernel MethodsKnowledge Representation and Reasoning: Computational Complexity of ReasoningComputer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsSearch and Retrieval-Augmented AI: Efficiency and scalability of Web search enginesSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

For years, many neural networks have been developed based on the Kolmogorov-Arnold Representation Theorem (KART), which was created to address Hilbert's 13th problem. Recently, relying on KART, Kolmogorov-Arnold Networks (KANs) have attracted attention from the research community, stimulating the use of polynomial functions such as B-splines and RBFs. However, these functions are not fully supported by GPU devices and are still considered less popular. In this paper, we propose the use of fast computational functions, such as ReLU and trigonometric functions (e.g., ReLU, sin, cos, arctan), as basis components in Kolmogorov-Arnold Networks (KANs). By integrating these function combinations into the network structure, we aim to enhance computational efficiency. Experimental results show that these combinations maintain competitive performance while offering potential improvements in training time and generalization.

Problem

• Proposing efficient activation functions for Kolmogorov-Arnold Networks
• Enhancing computational efficiency through fast function combinations
• Maintaining performance while improving training time and generalization

Innovation

• ReLU and trigonometric functions as basis
• Enhancing computational efficiency in KANs
• Maintaining competitive performance with faster training