Effective Dimension Aware Fractional-Order Stochastic Gradient Descent for Convex Optimization Problems

2025-03-17

AI Summary

Manual tuning of the fractional order exponent in Fractional-order Stochastic Gradient Descent (FOSGD) is challenging and often leads to training instability. Method: We propose 2SED-FOSGD, an adaptive fractional-order optimization algorithm leveraging the Two-Scale Effective Dimension (2SED) framework. It introduces, for the first time, a data-driven effective dimensionality awareness mechanism into fractional-order optimization, enabling online, dynamic calibration of the fractional order exponent—thereby preserving long-range memory benefits while ensuring convergence stability. Contribution/Results: Theoretical analysis establishes convergence guarantees for convex objectives. Empirical evaluation across multiple benchmark tasks demonstrates significant improvements in convergence speed and final accuracy, alongside enhanced robustness and generalization capability—particularly advantageous in large-scale machine learning settings.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Fractional-order stochastic gradient descent (FOSGD) leverages a fractional exponent to capture long-memory effects in optimization, yet its practical impact is often constrained by the difficulty of tuning and stabilizing this exponent. In this work, we introduce 2SED Fractional-Order Stochastic Gradient Descent (2SEDFOSGD), a novel method that synergistically combines the Two-Scale Effective Dimension (2SED) algorithm with FOSGD to automatically calibrate the fractional exponent in a data-driven manner. By continuously gauging model sensitivity and effective dimensionality, 2SED dynamically adjusts the exponent to curb erratic oscillations and enhance convergence rates. Theoretically, we demonstrate how this dimension-aware adaptation retains the benefits of fractional memory while averting the sluggish or unstable behaviors frequently observed in naive fractional SGD. Empirical evaluations across multiple benchmarks confirm that our 2SED-driven fractional exponent approach not only converges faster but also achieves more robust final performance, suggesting broad applicability for fractional-order methodologies in large-scale machine learning and related domains.

Problem

• Automatically calibrates fractional exponent in FOSGD
• Enhances convergence rates and stability in optimization
• Improves performance in large-scale machine learning tasks

Innovation

• Combines Two-Scale Effective Dimension with FOSGD
• Automatically calibrates fractional exponent data-driven
• Enhances convergence rates and stabilizes optimization