Fast and Scalable Caputo Fractional Gradient Descent via Perturbation-Preserving Memory Compression

2026-07-16

AI Summary

This work addresses the impractical computational complexity of Caputo fractional-order gradient descent, which scales quadratically with the number of iterations due to its history dependence. The authors reformulate the fractional gradient as a discrete convolution of past gradients and introduce two complementary memory compression mechanisms: an exponential sum (SOE) approximation of the power-law kernel enabling recursive updates, and a novel dyadic hierarchical discrete convolution (DHDC) for multiscale gradient aggregation. Viewing memory compression as a controlled perturbation of the ideal Caputo operator, the approach preserves the long-term memory structure while maintaining favorable optimization dynamics. Under μ-strong convexity and L-smoothness assumptions, the compressed algorithm is theoretically guaranteed to retain monotonic decrease of the objective function and linear convergence, achieving substantial computational savings without compromising convergence properties.

Abstract

Fractional gradient descent (FGD) incorporates long-range memory through Caputo-type operators and has been shown to improve stability in ill-conditioned and nonconvex optimization problems. Despite these advantages, its practical use remains limited, mainly due to the high computational cost of evaluating history-dependent convolutions, which scales quadratically with the number of iterations. In this paper, we focus on making Caputo-based optimization computationally viable without sacrificing its intrinsic memory structure. We begin by expressing the fractional descent direction as a discrete convolution over past gradients, which provides a unified view of the method. Based on this formulation, we introduce two complementary mechanisms to reduce the cost of the memory term. The first uses a sum-of-exponentials (SOE) approximation of the power-law kernel, leading to efficient recursive updates. The second approach, newly proposed in this paper as dyadic hierarchical discrete convolution (DHDC), compresses the gradient history through a multiscale aggregation strategy. Rather than treating these approximations as purely numerical accelerations, we interpret them as perturbations of the ideal Caputo operator. This viewpoint allows us to analyze how the compressed memory affects the optimization dynamics. Under standard $μ$-strong convexity and $L$-smoothness assumptions, we show that the resulting method still exhibits monotone descent and linear convergence, provided that the approximation error remains controlled.

Problem

• fractional gradient descent
• Caputo operator
• memory compression
• computational cost
• history-dependent convolution

Innovation

• fractional gradient descent
• memory compression
• sum-of-exponentials approximation
• dyadic hierarchical discrete convolution
• Caputo operator