🤖 AI Summary
Current deep learning frameworks lack autograd-compatible fractional-order matrix differentiation, hindering the integration of fractional calculus into neural network optimization. To address this, we propose the fractional-order Jacobian matrix differential ( J^alpha ) and establish a matrix-form fractional chain rule compatible with automatic differentiation, enabling end-to-end differentiable propagation of fractional-order gradients through hidden layers. Leveraging PyTorch, we design a Riemann–Liouville-based fractional linear layer (FLinear) that seamlessly integrates into standard backpropagation. Experiments on multilayer perceptrons demonstrate that FLinear significantly reduces training loss and improves test accuracy, while maintaining reasonable GPU memory consumption and training efficiency. This work provides both a theoretical foundation and a practical implementation for incorporating fractional calculus into differentiable programming paradigms for deep learning.
📝 Abstract
Fractional-order differentiation has many characteristics different from integer-order differentiation. These characteristics can be applied to the optimization algorithms of artificial neural networks to obtain better results. However, due to insufficient theoretical research, at present, there is no fractional-order matrix differentiation method that is perfectly compatible with automatic differentiation (Autograd) technology. Therefore, we propose a fractional-order matrix differentiation calculation method. This method is introduced by the definition of the integer-order Jacobian matrix. We denote it as fractional-order Jacobian matrix differentiation (${{f{J}}^alpha }$). Through ${{f{J}}^alpha }$, we can carry out the matrix-based fractional-order chain rule. Based on the Linear module and the fractional-order differentiation, we design the fractional-order Autograd technology to enable the use of fractional-order differentiation in hidden layers, thereby enhancing the practicality of fractional-order differentiation in deep learning. In the experiment, according to the PyTorch framework, we design fractional-order Linear (FLinear) and replace nn.Linear in the multilayer perceptron with FLinear. Through the qualitative analysis of the training set and validation set $Loss$, the quantitative analysis of the test set indicators, and the analysis of time consumption and GPU memory usage during model training, we verify the superior performance of ${{f{J}}^alpha }$ and prove that it is an excellent fractional-order gradient descent method in the field of deep learning.