More Optimal Fractional-Order Stochastic Gradient Descent for Non-Convex Optimization Problems

2025-05-05

AI Summary

To address the challenges of hyperparameter tuning difficulty, severe oscillations, and slow convergence in fractional-order stochastic gradient descent (FOSGD) for nonconvex optimization—stemming from its fixed fractional-order exponent—this paper proposes the 2SEDFOSGD algorithm. The method introduces, for the first time, a dual-scale effective dimension (2SED)-aware mechanism that enables data-driven, dynamic, and adaptive adjustment of the fractional-order exponent. By integrating fractional calculus, α-stable noise modeling, and dual-scale sensitivity analysis, 2SEDFOSGD significantly enhances robustness and convergence efficiency. Empirical evaluation on autoregressive (AR) models under both Gaussian and α-stable noise demonstrates that 2SEDFOSGD achieves a 37% faster convergence rate and reduces parameter estimation error by 29%, outperforming standard SGD, Adam, and conventional FOSGD.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Fractional-order stochastic gradient descent (FOSGD) leverages fractional exponents to capture long-memory effects in optimization. However, its utility is often limited by the difficulty of tuning and stabilizing these exponents. We propose 2SED Fractional-Order Stochastic Gradient Descent (2SEDFOSGD), which integrates the Two-Scale Effective Dimension (2SED) algorithm with FOSGD to adapt the fractional exponent in a data-driven manner. By tracking model sensitivity and effective dimensionality, 2SEDFOSGD dynamically modulates the exponent to mitigate oscillations and hasten convergence. Theoretically, for onoconvex optimization problems, this approach preserves the advantages of fractional memory without the sluggish or unstable behavior observed in na""ive fractional SGD. Empirical evaluations in Gaussian and $alpha$-stable noise scenarios using an autoregressive (AR) model highlight faster convergence and more robust parameter estimates compared to baseline methods, underscoring the potential of dimension-aware fractional techniques for advanced modeling and estimation tasks.

Problem

• Adapting fractional exponents in FOSGD for stability
• Mitigating oscillations in non-convex optimization problems
• Improving convergence speed in stochastic gradient descent

Innovation

• Integrates 2SED algorithm with FOSGD
• Dynamically modulates fractional exponent
• Enhances convergence and stability