🤖 AI Summary
Traditional learning rate schedulers suffer from sensitivity to training duration (number of epochs), poor cross-task transferability, and insufficient robustness under resource constraints. To address these issues, this work proposes two novel analytical schedulers—HyperbolicLR and ExpHyperbolicLR—that leverage the asymptotic properties of hyperbolic functions to construct epoch-agnostic scheduling mechanisms, enabling seamless hyperparameter reuse across varying training lengths. We further introduce a two-phase strategy: rapid hyperparameter tuning in early epochs followed by fixed-rate scheduling in later epochs—balancing convergence speed and optimization stability. Extensive experiments on image classification, time-series forecasting, and operator learning demonstrate that our methods significantly outperform baselines—including StepLR and CosineAnnealing—in long-horizon training, achieving superior performance stability and generalization. Results confirm enhanced efficiency and robustness under computational constraints.
📝 Abstract
This study proposes two novel learning rate schedulers -- Hyperbolic Learning Rate Scheduler (HyperbolicLR) and Exponential Hyperbolic Learning Rate Scheduler (ExpHyperbolicLR) -- to address the epoch sensitivity problem that often causes inconsistent learning curves in conventional methods. By leveraging the asymptotic behavior of hyperbolic curves, the proposed schedulers maintain more stable learning curves across varying epoch settings. Specifically, HyperbolicLR applies this property directly in the epoch-learning rate space, while ExpHyperbolicLR extends it to an exponential space. We first determine optimal hyperparameters for each scheduler on a small number of epochs, fix these hyperparameters, and then evaluate performance as the number of epochs increases. Experimental results on various deep learning tasks (e.g., image classification, time series forecasting, and operator learning) demonstrate that both HyperbolicLR and ExpHyperbolicLR achieve more consistent performance improvements than conventional schedulers as training duration grows. These findings suggest that our hyperbolic-based schedulers offer a more robust and efficient approach to deep network optimization, particularly in scenarios constrained by computational resources or time.